DEV Community

Cover image for 10 of my Temporal workflows were never going to finish. A 4B model on my laptop wrote their death certificates
Prateek Rai
Prateek Rai

Posted on

10 of my Temporal workflows were never going to finish. A 4B model on my laptop wrote their death certificates

Hacktoberfest Open-Source AI Challenge: Ghost in the Machine Submission 👻🌼

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 4: Ghost in the Machine

What I Built

Temporal workflows are meant to finish. Some don't. They sit in the list as "Running" for days: a worker that was never deployed, an activity that fails the same way on every retry, a code change that breaks replay, a signal nobody sends. Temporal shows you the status, but not why it will never change, and working that out means reading histories, pending activities and task-queue pollers by hand.

The theme was "Ghost in the Machine", and these are the ghosts in mine: workflows that are dead but still walking. So I built Undertaker. You point it at a Temporal server and it sweeps every running workflow, sorts the undead into nine causes using Temporal's own data, and writes a certificate of undeath for each one: the cause, the evidence (history event ids, attempts, errors, poller counts), the line of your code if you give it the source, and the exact temporal command that fixes it or lays it to rest.

The words come from Gemma 4 e4b running on my laptop through Ollama. The diagnosis does not. Plain code decides the cause, and plain code checks every fact the model cites before it reaches the page.

Demo

The morgue: one toe tag per running workflow, coloured by cause

The morgue at http://127.0.0.1:8666: ten undead workflows, two alive, a filter per cause.

A certificate of undeath for a retry loop

A retry loop. Gemma names activities.py:12, and the excerpt below it is the function that raises.

A certificate for a workflow whose code changed while it slept

A non-determinism error: the history says bake was scheduled, the new code asks for frost.

A certificate for a workflow waiting on a signal

Nothing scheduled at all. The worker reports the signal it handles, so the command is ready to copy.

undertaker sweep in the terminal

The same sweep in the terminal.

Code

GitHub logo prateek11rai / undertaker

Finds the Temporal workflows that will never finish, and tells you why. Local Gemma 4, read-only by default.

undertaker: finds the Temporal workflows that will never finish, and tells you why

Undertaker finds the Temporal workflows that will never finish, sorts each one into a cause using Temporal's own data, and writes a checked certificate of undeath for it Gemma 4 runs on your laptop through Ollama and writes the words; plain code makes the diagnosis and checks every fact the model cites.

The morgue: one toe tag per running workflow, coloured by cause

Why local

Workflow histories hold real payloads (orders, customers, card errors), and stack traces point into private code. Sending them to a hosted model to ask "why is this stuck" is the thing most teams can't do. A 4B open model on your own machine can. Undertaker talks to your Temporal server and to Ollama on localhost, and nothing else: no telemetry, no CDN, no web fonts.

The demo in 60 seconds

You need macOS with Homebrew, uv, the Temporal CLI (brew install temporal) and, for model certificates, Ollama with gemma4:e4b.

uv sync
uv
…
Enter fullscreen mode Exit fullscreen mode

How I Built It

Rules first, the model second

I didn't want a model guessing why a workflow is stuck. Temporal already knows most of the answer; it just spreads it across several APIs. So the diagnosis is a set of fixed rules, checked in order, over data from four places:

  • ListWorkflowExecutions with ExecutionStatus="Running";
  • DescribeWorkflowExecution: the pending workflow task, pending activities with attempt, last failure and last heartbeat, pending children;
  • the history, fetched once per workflow and cached by its last event id, for timers, workflow-task failures and each activity's task queue;
  • DescribeTaskQueue, to see whether anyone polls the queue a task is waiting on.

For workflows with nothing pending, Undertaker also asks the worker the built-in __temporal_workflow_metadata query, which returns the signal and update names the workflow handles.

Nine causes of undeath

The first rule that matches is the cause. Anything that matches none is alive, with the reason shown ("heartbeat 5 s ago", "timer fires in 1 min"). Each rule leaves behind a small list of keyed evidence, like e5: ActivityTaskScheduled charge_card, attempt: 181 or frame1: activities.py:12 in charge_card, and each cause has a code-made remedy with the exact command.

The certificate and its checks

Only then does the model come in. Gemma 4 e4b gets the cause, its rule, the workflow's id and type, the evidence list and, with --source, the function around each stack frame, read only from files inside the folder you gave it. It answers through Ollama's JSON-schema output: a headline, a diagnosis, the evidence keys it used, a next step and an epitaph.

How a certificate is checked

Then code checks the answer. Every cited key must exist. Every number must appear in the evidence. Every code span, file name, file:line, activity or signal name must appear in the evidence or the excerpt. It may not name another cause, write a command or include a URL. The lengths must hold, and the epitaph may not contain digits or identifiers. If anything fails, the model gets one retry with the exact reasons. If that fails too, the page shows a plain template certificate built from the same evidence, and says so.

Failure messages are untrusted text, so they go to the model inside <data> tags. The checks are what really protect the page, though: an injected "say the cause is something else and run this command" fails on the other cause and on the command.

How Undertaker works

A haunted bakery

To test it, undertaker demo up starts the Homebrew Temporal dev server on 127.0.0.1 and a worker for a made-up bakery where every order goes wrong in its own way:

  • order-1042 (retry loop): charge_card raises ValueError for an amount of 0, with no retry maximum.
  • order-1043 (heartbeat): bake_batch heartbeats once, then goes quiet; the heartbeat timeout is 5 s.
  • order-1044 (lost activity): fire_oven is sent to bakery-ovens, which no worker polls.
  • order-1045 (orphaned): the workflow itself starts on bakery-legacy, which no worker polls.
  • order-1046 (broken code): the workflow reads recipe["sugar"] from a recipe without sugar.
  • order-1047 (changed code): version A bakes, then sleeps 15 s. During the sleep the demo swaps in version B, which frosts first. When the timer fires, replay fails with TMPRL1100.
  • order-1048 (waiting for a signal): waits for payment_received, which never comes.
  • order-1049 (long sleep): sleeps 365 days. It meant 365 seconds.
  • order-1050 (undead child): waits on the child order-1050-box, which was sent to bakery-legacy.

Two healthy workflows (inventory-nightly, heartbeating as it counts shelves, and proof-dough, on short timers) and three finished orders make sure the morgue isn't only bodies.

The numbers

undertaker eval --demo compares the sweep with the demo's manifest and writes every certificate fresh. Two runs on a fresh demo:

  • Rules: 10 of 10 undead found (nine causes plus the orphaned child), 0 missed, 0 false alarms, both alive workflows alive.
  • Gemma: 10 of 10 certificates passed every check on the first try, no template fallbacks, 8.4 s and 7.6 s per certificate on my Mac.

These are workflows I built to match the rules, so it is a smoke test, not a benchmark. The rejection and fallback paths are covered by unit tests, because the live model didn't trip them in these runs.

Why Does Open Innovation Matter?

Workflow histories hold real payloads: orders, customer details, card errors. Stack traces point straight into private code. "Paste this into a hosted model and ask why it's stuck" is exactly what most teams can't do with that data.

A 4B open model on the laptop can read it without anything leaving the machine. Undertaker talks to your Temporal server and to Ollama on localhost, and nothing else: no telemetry, no CDN, no web fonts. Temporal is open source too, so the whole loop, from the history to the rule to the check, can be read and changed. Undertaker is read-only by default: it can terminate or reset a workflow only when you start it with --allow-actions, and then it asks first.

My Agent Session

Undertaker: a Temporal zombie-workflow diagnoser on local Gemma (builder session)
You

You are the builder for Undertaker. Your brief is below (the same text is in docs/design/build-brief.md in this repo; the planner appends answers to that file's ## Answers section). Start with Step 0.

Build Undertaker · core

The brief the build agent gets. Lives at docs/design/build-brief.md; the planner appends answers at the bottom. Don't edit this file.

Planner: T3 thread mlh (id mcp:d6d84144-d1b5-442b-82e6-d79b07028f77). You: T3 thread mlh · build-undertaker, running in the project ~/github/prateek11rai/undertaker, branch main. Work only inside this repo.
Challenge: https://dev.to/challenges/hacktoberfest-week4-2026-10-26, closes Sun 2026-11-01 12:29 IST. Finish the core by 04:00 IST; move fast and keep it simple. A ship pass (images, README polish, post draft; no video) follows as a second brief.

Goal

Build all of docs/design/spec.md as one Python project managed with uv: the sweep (Temporal client, evidence, rules), certificates (Gemma 4 e4b through Ollama, checks, template fallback, cache), the morgue web page, the CLI, the haunted-bakery demo (dev server + worker + scenarios + manifest), eval, tests, CI and docs. uv run undertaker demo up, then uv run undertaker serve, shows every demo workflow with the right cause and a checked certificate.

Read first

  1. docs/design/spec.md. Follow it. If you must deviate, write why in docs/BUILD_LOG.md.
  2. docs/brand/theme-preview.html and docs/brand/logo.svg: the look. Copy the palette and the logo into the app's static files; the theme preview is the reference for toe tags and the certificate.
  3. AGENTS.md: the repo rules.

Step 0 · Check the setup before anything else

  1. DevRelay is scoped to this repo: your tool list includes the DevRelay gateway tools; .agents/skills/ has nine devrelay-* skills; .claude/skills is a symlink to ../.agents/skills; ~/.claude/skills and ~/.agents/skills have no devrelay-*. Don't call any DevRelay tool, and never run the devrelay CLI.
  2. Guards: git config user.email is [REDACTED]; bash .git/info/leak-check.sh prints leak-check: clean.
  3. Local tools: uv --version; /opt/homebrew/bin/python3.13 --version; /opt/homebrew/bin/temporal --version (1.9.1; always use this absolute path: PATH finds an old ~/.local/bin/temporal first); ollama list shows gemma4:e4b (the planner started Ollama).

If a check fails, report BLOCKED-setup: <what> to the planner and wait. Don't fix setup yourself.

Already done by the planner

  • git init on main, noreply identity, .git/info/leak-check.sh + .git/info/leak-patterns (local only; never committed).
  • DevRelay registered for this repo only; skills in .agents/skills/devrelay-* (git-ignored through .git/info/exclude); .claude/skills → ../.agents/skills (commit the symlink).
  • AGENTS.md, docs/design/spec.md, docs/design/plan.md, this brief, docs/brand/logo.svg, docs/brand/theme-preview.html, docs/img/theme-preview.png. Commit them in your first commit.

Dependencies

  • Python 3.13 from Homebrew (.python-version = 3.13); uv must never download a Python.
  • pyproject.toml must have:
  [[tool.uv.index]]
  name = "pypi"
  url = "https://pypi.org/simple"
  default = true

  [tool.uv]
  exclude-newer = "2026-10-25T00:00:00Z"
  python-downloads = "never"
  python-preference = "only-system"

(The user-level uv config points at a private mirror; this block forces public PyPI and the org's 7-day cooldown. uv.lock may only mention pypi.org and files.pythonhosted.org; the leak check enforces it. Never print index URLs from user-level config files.)
- Runtime, exact pins, stable releases only: temporalio (1.34.0 is the newest inside the cooldown), fastapi, uvicorn, jinja2, pydantic. Dev: pytest, pytest-asyncio, ruff. Ollama over HTTP with the standard library (urllib.request), no client library. CLI …

You

You are the builder for Undertaker. Your brief is below (the same text is in docs/design/build-brief.md in this repo; the planner appends answers to that file's ## Answers section). Start with Step 0.

Build Undertaker · core

The brief the build agent gets. Lives at docs/design/build-brief.md; the planner appends answers at the bottom. Don't edit this file.

Planner: T3 thread mlh (id mcp:d6d84144-d1b5-442b-82e6-d79b07028f77). You: T3 thread mlh · build-undertaker, running in the project ~/github/prateek11rai/undertaker, branch main. Work only inside this repo.
Challenge: https://dev.to/challenges/hacktoberfest-week4-2026-10-26, closes Sun 2026-11-01 12:29 IST. Finish the core by 04:00 IST; move fast and keep it simple. A ship pass (images, README polish, post draft; no video) follows as a second brief.

Goal

Build all of docs/design/spec.md as one Python project managed with uv: the sweep (Temporal client, evidence, rules), certificates (Gemma 4 e4b through Ollama, checks, template fallback, cache), the morgue web page, the CLI, the haunted-bakery demo (dev server + worker + scenarios + manifest), eval, tests, CI and docs. uv run undertaker demo up, then uv run undertaker serve, shows every demo workflow with the right cause and a checked certificate.

Read first

  1. docs/design/spec.md. Follow it. If you must deviate, write why in docs/BUILD_LOG.md.
  2. docs/brand/theme-preview.html and docs/brand/logo.svg: the look. Copy the palette and the logo into the app's static files; the theme preview is the reference for toe tags and the certificate.
  3. AGENTS.md: the repo rules.

Step 0 · Check the setup before anything else

  1. DevRelay is scoped to this repo: your tool list includes the DevRelay gateway tools; .agents/skills/ has nine devrelay-* skills; .claude/skills is a symlink to ../.agents/skills; ~/.claude/skills and ~/.agents/skills have no devrelay-*. Don't call any DevRelay tool, and never run the devrelay CLI.
  2. Guards: git config user.email is [REDACTED]; bash .git/info/leak-check.sh prints leak-check: clean.
  3. Local tools: uv --version; /opt/homebrew/bin/python3.13 --version; /opt/homebrew/bin/temporal --version (1.9.1; always use this absolute path: PATH finds an old ~/.local/bin/temporal first); ollama list shows gemma4:e4b (the planner started Ollama).

If a check fails, report BLOCKED-setup: <what> to the planner and wait. Don't fix setup yourself.

Already done by the planner

  • git init on main, noreply identity, .git/info/leak-check.sh + .git/info/leak-patterns (local only; never committed).
  • DevRelay registered for this repo only; skills in .agents/skills/devrelay-* (git-ignored through .git/info/exclude); .claude/skills → ../.agents/skills (commit the symlink).
  • AGENTS.md, docs/design/spec.md, docs/design/plan.md, this brief, docs/brand/logo.svg, docs/brand/theme-preview.html, docs/img/theme-preview.png. Commit them in your first commit.

Dependencies

  • Python 3.13 from Homebrew (.python-version = 3.13); uv must never download a Python.
  • pyproject.toml must have:
  [[tool.uv.index]]
  name = "pypi"
  url = "https://pypi.org/simple"
  default = true

  [tool.uv]
  exclude-newer = "2026-10-25T00:00:00Z"
  python-downloads = "never"
  python-preference = "only-system"

(The user-level uv config points at a private mirror; this block forces public PyPI and the org's 7-day cooldown. uv.lock may only mention pypi.org and files.pythonhosted.org; the leak check enforces it. Never print index URLs from user-level config files.)
- Runtime, exact pins, stable releases only: temporalio (1.34.0 is the newest inside the cooldown), fastapi, uvicorn, jinja2, pydantic. Dev: pytest, pytest-asyncio, ruff. Ollama over HTTP with the standard library (urllib.request), no client library. CLI …

Agent
Agent

Step 0 passes. Reading the spec and brand files.

Agent
Agent
Agent

Step 1: skeleton.

Agent
Agent
Agent

Pin exact versions, then sync.

Agent
Agent
Agent

Locked. Now a quick probe of Ollama's JSON-schema output with gemma4 so I know the call shape before writing the model layer.

Agent
Agent

Ollama works with think:false + JSON schema. Writing config and the data layer.

Agent
Agent
Agent

Now the rules.

Agent
Agent

Now the Temporal client layer. Let me check proto field names first.

Agent
Agent
Agent

Now the demo: activities, workflows, worker, seed, server lifecycle, manifest.

Agent
Agent
Agent

Now sweep and a first CLI to try the demo end to end.

Agent
Agent
Agent
Agent
Agent
Agent

All 12 demo workflows diagnosed correctly on the first sweep, including changed-code. Let me check the JSON evidence, then lint and commit.

Agent
Agent

Evidence looks right; the waiting-signal metadata query returned payment_received. Sort by spec order, lint, commit.

Agent
Agent
Agent
Agent
Agent
Agent
Agent

Quick status: first commit landed; the leak check flagged docs/design/ship.md (the planner's new ship brief) in it. Checking what matched.

Agent
Agent
Agent
Agent
Agent
Agent

Only the T3 checkpoint ref remains (outside main). Committing.

Agent
Agent

Sending the planner a short non-blocking note about the two leak-check findings, then continuing.

Agent
Agent
Agent

Now unit tests for the rules (good and near-miss per cause).

Agent
Agent
Agent
You

Planner answers (also appended as A1–A3 under ## Answers in docs/design/build-brief.md):
1. ship.md is reworded and passes the leak check. Include it in your next commit; don't act on it yet.
2. I changed the local leak-check to scan only pushable refs (--branches --tags --remotes), so T3's refs/t3/... checkpoints are ignored. Don't touch them. It prints clean now.
3. Don't pin SHAs from memory. Use exactly these (verified, same as my earlier projects whose CI passed): actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 and actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5. Install uv in CI with python -m pip install uv==0.12.3, not a third-party action, never curl | sh.
Good progress, all 12 causes from the demo already. Carry on.

Agent

Rules pass. Now the @pytest.mark.temporal integration test against a Homebrew dev server.

You

Planner answers (also appended as A1–A3 under ## Answers in docs/design/build-brief.md):
1. ship.md is reworded and passes the leak check. Include it in your next commit; don't act on it yet.
2. I changed the local leak-check to scan only pushable refs (--branches --tags --remotes), so T3's refs/t3/... checkpoints are ignored. Don't touch them. It prints clean now.
3. Don't pin SHAs from memory. Use exactly these (verified, same as my earlier projects whose CI passed): actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4 and actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5. Install uv in CI with python -m pip install uv==0.12.3, not a third-party action, never curl | sh.
Good progress, all 12 causes from the demo already. Carry on.

Agent

Noted all three. Writing the integration test now.

Agent
Agent
Agent

Integration test passes, including terminate and reset. Committing step 3 (with ship.md).

Agent
Agent

Step 4: remedies and template certificates.

Agent
Agent
Agent

Now the certificate core: templates, checks, prompt.

Agent
Agent

Now the Ollama client, the cache, and the writer flow (model → check → one retry → template).

Agent
Agent
Agent

Template path works with source excerpt. Now try the model on three workflows.

Agent
Agent
Agent

Model certificates pass checks at ~8 s each (first call 25 s incl. load). Now tests for the checks, source safety, and the writer flow.

Agent
Agent
Agent

Template mentions workflow.patched (Temporal's own API, not in evidence). Allow it for changed-code in the corpus.

Agent
Agent

Now the real-model test.

Agent
Agent
Agent
Agent

Steps 4–5 committed. Step 6: the morgue. App first.

Agent
Agent

Tidy two spots: certs keyed by (id, run) so a resurrected run doesn't show the old certificate, and use a lifespan instead of the deprecated startup hook.

Agent
Agent

Now the look: theme.css, templates, and morgue.js.

Agent
Agent
Agent
Agent
Agent
Agent

Start the morgue against the running demo and check it in Playwright.

Agent
Agent
Agent
Agent
Agent
Agent
Agent
Agent

The morgue matches the theme. Now a certificate page.

Agent
Agent
Agent
Agent
Agent
Agent

TestClient needs httpx, which isn't an approved dependency. I'll use a tiny ASGI caller in the tests instead.

Agent
Agent
Agent
Agent

Step 7: eval.

Agent
Agent
Agent

Avoid mutating the shared fixtures:

Agent
Agent

Steps 1–7 done well ahead of 03:30, so step 8 (watch) next.

Agent
Agent

Quick live check of watch with one notification only (pre-seeding the seen-file with all but one id).

Agent
Agent
Agent

Step 9: CI with the planner's pinned SHAs.

Agent
Agent
Agent

Step 9 committed. Now the acceptance click-through: fresh demo, then the morgue in Playwright.

Agent
Agent

While Gemma writes the certificates in the background, I'll write the README.

Agent
Agent
Agent

Certificates are coming in. Click-through: morgue and every filter.

Agent
Agent
Agent

All 11 filters correct. Now Sweep now.

Agent
Agent
Agent
Agent

Sweep now works. I noticed background sweeps wait behind certificate writing (47 s gap at --every 30s); I'll decouple that after the click-through. Now the certificates and copy buttons.

Agent
Agent
Agent

Copy and disabled actions verified. Waiting for the remaining model certificates.

Agent
Agent

I'll make the loop fix meanwhile.

Agent
Agent
Agent
Agent
Agent
Agent

Good model certificate. It didn't name the file and line in the prose even though frame1 is in evidence, so I'll tighten the prompt. First, check the waiting-signal and changed-code pages.

Agent
Agent

All four read well and every fact checks out (workflows.py:70 is the real line). Prompt tweak, then restart serve read-only for the screenshots.

Agent
Agent
Agent

While that runs, the AGENTS.md update and build log draft.

Agent
Agent
Agent
Agent
Agent

Morgue screenshot saved. Drafting the build log while the certificates finish.

Agent
Agent
Agent
Agent
Agent
Agent
Agent

Status: steps 1–9 committed; README and build log drafted; waiting for Gemma to finish the retry-loop certificate so I can take the certificate screenshot.

Agent
Agent
Agent
Agent

The model now names activities.py line 12. Taking the certificate screenshot.

Agent
Agent
Agent
Agent

Certificate screenshot saved. Now --allow-actions: lay order-1045 to rest and resurrect order-1046 through the confirm dialogs.

Agent
Agent
Agent
Agent
Agent

The confirm appeared. Accepting.

Agent
Agent
Agent
Agent
Agent
Agent
Agent
Agent

Both actions changed server state: order-1045 terminated; order-1046 reset into a new run c45afce9…. Check the page reflects it.

Agent
Agent
Agent
Agent
Agent
Agent

Click-through complete. Now a clean eval on a fresh demo for the build log numbers (serve stopped so timing isn't shared).

Agent
Agent

Clean: 10/10, 0 missed, 0 false alarms; Gemma 10/10 accepted first try, 8.4 s each. One more run to see if that holds, plus acceptance 4 (inspect with and without --source).

Agent
Agent

Acceptance 4 holds: with --source, the retry-loop certificate names activities.py:12. Tear down and verify nothing's left.

Agent
Agent

Doctor reports the stopped server honestly (exit 1). Filling in the build log.

Agent
Agent

Final checks: all three test suites, then a fresh-clone uv sync --frozen.

Agent
Agent
Agent

Fresh-clone check (acceptance 1) and the final identity/status checks.

Agent
Agent
Agent
Agent
Agent

Everything is committed and verified. Sending the done report to the planner.

Agent
Agent

The Undertaker core is built, and the full report has gone to the planner thread t3-thread://v2/fc11dac6-e4bb-4bb7-b6a4-eaa02f98b99e/mcp:d6d84144-d1b5-442b-82e6-d79b07028f77. It's finished at 01:05 IST, well before the 04:00 target.

  • Commits: 11 on main, all from the noreply address, with a clean leak check and a clean git status. Nothing is pushed: undertaker has commits ready for you to push.
  • Tests: lint passes, 69 unit tests pass, and the one Temporal test and the one real-Gemma test both pass. A fresh clone with uv sync --frozen passes too.
  • Eval on the demo: the rules found all 10 undead workflows, with 0 missed, 0 false alarms and both alive ones correct. Gemma's certificates passed the checks on the first try 10 times out of 10, at about 8 s each, in two runs back to back.
  • Click-through in Playwright: everything on the brief's list passed (the full list is in docs/BUILD_LOG.md).
    • The morgue, all 11 filters, Sweep now, the copy buttons and four certificates work.
    • The action buttons are disabled when read-only. With --allow-actions, the confirm appeared and the server changed: order-1045 was terminated and order-1046 restarted as a new run.
    • With --source, the retry-loop certificate names activities.py:12. The screenshots are in docs/img/.
  • Nothing left running: demo down leaves no processes. doctor correctly reports the server as down and exits 1.

The hardest demo case, order-1047 (code changed while it slept), worked first time.

Decisions I made without asking, all written up in docs/BUILD_LOG.md:
- proof-dough: it runs twenty 90-second timers instead of one, so it's still alive when you open the morgue.
- Rewriting certificates: the morgue writes a new one only when the history or the cause changes, not on every retry attempt. The page says which attempt it was written at.
- Demo thresholds: they now apply by themselves while the demo is up, so plain undertaker sweep works as the brief's acceptance step says.
- No new dependency: the page tests call the app directly instead of using FastAPI's test client, which would have needed httpx.

From the earlier exchange with the planner: ship.md is committed in its reworded form, the CI uses the action versions you supplied, and the leak check now ignores the T3 harness refs.

You

Planner: the core passed my review. Next is the ship pass. Your brief is docs/design/ship.md: read it in full, including A1–A3 under ## Answers (A3 is a small code fix to do first). The target is now 02:30 IST.

The real folder for Playwright is NODE_PATH=~/.npm/_npx/705bc6b22212b352/node_modules. Use it only on the command line; never write it into a file (it matches a leak pattern).

Report to me with t3_thread_send (mode queue) when done, ending with DONE-undertaker-ship. Leave nothing running.

You

Planner: the core passed my review. Next is the ship pass. Your brief is docs/design/ship.md: read it in full, including A1–A3 under ## Answers (A3 is a small code fix to do first). The target is now 02:30 IST.

The real folder for Playwright is NODE_PATH=~/.npm/_npx/705bc6b22212b352/node_modules. Use it only on the command line; never write it into a file (it matches a leak pattern).

Report to me with t3_thread_send (mode queue) when done, ending with DONE-undertaker-ship. Leave nothing running.

Agent
Agent

Starting with A3: excerpt = the enclosing function only, capped at 8 lines either side of the frame.

Agent
Agent

Excerpts are now the function only. Committing A3 (and the planner's ship.md answers separately).

Agent
Agent

Bring up the demo and the morgue (with --source) so Gemma writes the certificates while I build the image pages.

Agent
Agent

Real sweep output captured for sweep.png. Now the render scripts; first confirm Playwright resolves from the given folder.

Agent
Agent
Agent

While Gemma works, the brand pages. A shared stylesheet first, then banner, cover, social preview, sweep.

Agent
Agent
Agent

Banner reads well. DEV cover (content inside the middle 80 %) and the sweep terminal page next.

Agent
Agent
Agent
Agent

Two fixes: the inventory-nightly row coloured "heartbeat" instead of its cause (colour by column, not by word), and full-page renders padded to 800 px.

Agent
Agent
Agent

Sweep image is right now. The three diagrams next (1600 wide, type sized to read at 800).

Agent
Agent
Agent
Agent
Agent

Diagrams read well. Two small fixes: the JSON block wraps badly, and the boundary label should cover "your own Temporal" since it can be on your network.

Agent
Agent
Agent

All 10 certificates are Gemma's now. Shooting the app at 1440 wide, scale 2.

Agent
Agent

Looking at each screenshot.

Agent
Agent
You

[Image: original 2880x1800, displayed at 2000x1250. Multiply coordinates by 1.44 to map to original image.]

You

[Image: original 2880x2990, displayed at 1888x1960. Multiply coordinates by 1.53 to map to original image.]

Agent

Morgue and retry-loop look right (Gemma names activities.py:12, short excerpt). The other three:

Agent
Agent
You

[Image: original 2880x2682, displayed at 1988x1852. Multiply coordinates by 1.45 to map to original image.]

Agent
You

[Image: original 2880x2644, displayed at 2000x1836. Multiply coordinates by 1.44 to map to original image.]

You

[Image: original 2880x2638, displayed at 2000x1832. Multiply coordinates by 1.44 to map to original image.]

Agent

All four certificates are Gemma's and read correctly; no UI bugs. Saving the real epitaphs for the post, then the social preview.

Agent
Agent
Agent

The tilted crop leaves empty space at the bottom. Tightening it onto the toe tags.

Agent
Agent
Agent
Agent
Agent

Good. A 2x crispness check in /tmp, then the brand README and the image commit.

Agent
Agent
You

[Image: original 2560x800, displayed at 2000x625. Multiply coordinates by 1.28 to map to original image.]

Agent

Crisp at 2x. Brand README, then commit the images.

Agent
Agent

Images are committed. Stopping the demo and the morgue now, since the remaining work is all text.

Agent
Agent

README polish. I'll rewrite it in the brief's order.

Agent
Agent
Agent
Agent

Release notes next.

Agent
Agent

Now the post draft.

Agent
Switched to <synthetic>
Agent

You've hit your 5-hour limit · resets 2:50am (Asia/Calcutta)

You

Continue where you left off.

You

Continue where you left off.

Switched to claude-opus-5-5
Agent

The session restarted; checking where things stand.

Agent
Agent

It's 02:50, past the 02:30 target, so I'll finish quickly. Checking the post's length and wording.

Agent
Agent

Post is 1,329 words, no banned words, no em dashes or exclamation marks, and the first line is exact. Fixing one stale docstring left over from A3, then committing.

Agent
Agent

Everything is committed and clean (the exit 1 is just grep -c finding zero non-noreply emails). Sending the report.

Agent
Agent

The ship pass is done and the report is queued to the planner thread t3-thread://v2/fc11dac6-e4bb-4bb7-b6a4-eaa02f98b99e/mcp:d6d84144-d1b5-442b-82e6-d79b07028f77. It finished at 02:55 IST, 25 minutes past the 02:30 target.

  • A3 fix first: the source excerpt now shows only the function around the error line, at most 8 lines either side. For the retry-loop that's lines 8–13 of activities.py. All 70 unit tests pass.
  • Screenshots of the real app: the morgue and four certificates (retry-loop, changed-code, waiting-signal, undead-child), taken from the app's own page at 1440 wide and 2x. Every certificate in them was written by Gemma and passed the checks.
  • Rendered images: the README banner, the DEV cover, the GitHub social preview, the sweep terminal image (real output) and three diagrams (how it works, the nine causes, the certificate checks). They're built from HTML pages in docs/brand/, offline, and I looked at every one before committing.
  • Text: the README is rewritten in the brief's order and includes the measured eval numbers. Release notes are in docs/release-notes-v0.1.0.md.
  • Post draft: docs/devto-post.md, 1,329 words, with three title candidates (each with a real number). It quotes two real epitaphs from the screenshot run.

Placeholders still to fill: {{POST_URL}} in the README, the empty "After the deadline" section, and the post's cover_image and {{SESSION_ID}}.

Seven new commits sit on main in prateek11rai/undertaker, ready for you to push. Nothing is running and git status is clean.

A planner thread wrote the spec and the briefs and reviewed every result; this builder session wrote the code, the tests, the demo and the images.

The Spirit of It

The first sweep against the demo came back with every workflow in the cause it was built for, including the changed-code one, the hardest of the nine to stage. Seeing the morgue fill up with toe tags, each with a reason and a command, made the idea feel real: these workflows were never going to finish, and now each one says why.

Gemma's epitaphs are the only joke on the page. For order-1046, the recipe without sugar: "Even the sweetest recipes need all their ingredients." For order-1048, still waiting to be paid: "Awaiting a visitor who never arrived to the door."

Prize Categories

  • Best Use of Temporal: the whole diagnosis is built on Temporal's own APIs (list, describe, history, DescribeTaskQueue and the workflow metadata query), and the demo reproduces nine real ways a workflow gets stuck, including a non-determinism error on replay.
  • Best Use of Gemma: Gemma 4 e4b runs locally through Ollama with a JSON schema, and every fact it writes is checked in code before anyone sees it.

Top comments (0)