ChaosCypher v0.4.3: When a Clean Restart Lost More Work Than a Crash

v0.4.3 adds nothing new. No new features, no breaking API changes, no schema migrations: 54 commits since v0.4.2, 23 of them changing shipped code. Most of them are in the queue and the extraction pipeline, and most of those share one trait. The work was lost, counted twice, or quietly downgraded, and nothing reported it.

A clean restart was worse than kill -9

Stopping a worker gracefully destroyed work that a hard kill would have recovered.

On shutdown, the drain's CancelledError arm wrote terminal status="cancelled" to the tasks it interrupted. That is exactly the status the requeue script and retry_task refuse to resurrect. So the recovery contract ran backwards: a SIGKILL left the task recoverable, and a polite restart finished it off.

It was one of four queue-recovery defects in this release that each looked like a working system:

• Crash recovery spent two units of the retry budget per cycle. requeue_atomic.lua bumped attempts on top of the worker's own claim-time increment, so a max_tries=5 task with retry_on_crash=True got 3 dispatches, not 5, and then went terminal reporting "crashed after 5 attempts". The affected operations are the four crash-retryable ones with no owning resource: execute_workflow, execute_step, export_graph, export_by_sources. For those, the queue budget is the only recovery layer.
• A lapsed heartbeat could never be refreshed. refresh_heartbeat issued an EXPIRE, which is a silent no-op on a key that has already expired. A 31-second stall condemned a multi-hour task as abandoned, and the reconciler requeued it for duplicate execution. It is now a SET with EX.
• A re-delivered chunk task could finish the job one chunk early. The short-circuit that made re-running an extraction chunk safe only caught a completed row. A row still in running went straight through, so two deliveries produced two terminal writes and two job-counter bumps. Both transitions are now guarded updates that return a rowcount, and the handler bails out before the billable LLM call when it loses the claim.

In plain English: if you restart workers, or your machine stalls for half a minute, queued work now comes back the number of times it was promised to, and only once.

Graph edges that explained themselves by thinking out loud

Every relationship ChaosCypher extracts carries a justification: the evidence the model gave for linking two entities. You can inspect it, and it is part of why the graph is checkable rather than taken on trust. That only works if the justification is evidence.

Some weren't. One parent_of edge at confidence 0.9 explained itself with "I will link 9 to 3 via interacts_with". That is the model deliberating, stored as if it were the reason the edge exists. Justifications that carry reasoning markers are now blanked, and the rest are trimmed to two sentences on a sentence or word boundary, so no fragment survives.

In plain English: when you read why an edge exists, you now get a reason or nothing, not the model's scratchpad.

One MCP sandbox bypass, and a read-only mode that wasn't

MCP add_document skipped its sandbox containment whenever content was supplied. The tool schema says that supplying content means the file is never loaded. Nothing implemented that: file_path was read from disk on every path and reached the loader uncontained. A caller supplying both got an arbitrary-file read that reached /data/credentials.json. The containment and dotfile guards are now unconditional, in the server and in the processor's mirror of it.

chaoscypher mcp honoured a read-only install only when you typed the flag. The guard read the --mode flag rather than the effective setting, so the documented Claude Desktop invocation against a read-configured install still handed the external client the destructive apply_upgrade tool. The guard now reads the configured mode, and an explicit --mode write still overrides it.

Also closed: the API-key selector and bearer-token throttle defects found in a post-merge sweep; decode_session rejects non-ASCII input; summarize gets the untrusted-document fence its five sibling handlers already had; and CHAOSCYPHER_ALLOW_USER_PLUGINS=0, the documented Docker kill switch for user plugins, now actually reaches all five plugin-loading services. Before, no service declared it, so the loader always saw its default.

In plain English: if you point an AI assistant at ChaosCypher over MCP, it can no longer read files outside the sandbox, and "read-only" means read-only however you launch it.

Settings that silently did nothing

• Every configured trigger filter matched everything. list_triggers deferred the filters and actions columns the dispatcher reads, so a trigger scoped to one source fired on every source. The .ccx exporter and GET /workflows/{id}/triggers were losing the same keys.
• A rejected workers.yaml value collapsed the Operations queue from 8 concurrent slots to 1, because a bad value fell through to a boolean instead of the worker type's default.
• The commit-time orphan filter ignored the source's own settings and always dropped orphans, even for minimal or unfiltered sources and rows with protect_orphans=True.
• The queue reset promised cancellation and delivered dead letters. It now runs cancel_all_tasks first, so running tasks stop instead of landing in the dead-letter set.

The API stops freezing on Cancel All

POST /queue/cancel-all ran a synchronous connect → SELECT → commit → disconnect on the event loop, once per task. After a large import that is thousands of cycles inside one request with no awaits, and Cortex served nothing else in the meantime. All four call sites now go through a worker thread. The same treatment went to per-chunk progress writes on the worker. A commit that lost the SQLite writer lock could park the loop for up to 60 seconds and starve the heartbeat refresher, which is how live tasks started to look abandoned.

The vision-page fan-out went from up to 4000 sequential round trips (bounded by vision_max_pages, default 2000) to 2. The changelog has the rest of the round: projections and unprojected reads fixed across the stuck-source sweep, step progress, queue stats and chaoscypher source search. These describe what the code was doing, not a benchmark.

Upgrading

No migrations apply. Drain the queue before you swap the image: stop new submissions and wait for /api/v1/queue/stats to report 0 pending on all queues. Don't run mixed versions against the same queue.

docker pull ghcr.io/chaoscypherinc/chaoscypher:0.4.3

Or, if you run the Python packages directly:

pip install -U chaoscypher-core chaoscypher-cortex chaoscypher-neuron chaoscypher-cli

Starting fresh:

docker run -d --name chaoscypher \
  -p 80:80 \
  -p 443:443 \
  -v chaoscypher-data:/data \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/chaoscypherinc/chaoscypher:latest

Next steps

1. If you use MCP with a read-configured install, upgrade first. That is the fix with the widest reach.
2. If you set trigger filters, check your automations after upgrading. Triggers you scoped to one source now fire only on that source.

Full details in the changelog. ChaosCypher is AGPL-3.0 and local-first: the graph, the chat, and the import and export paths all run on your own machine. Repo: https://github.com/chaoscypherinc/chaoscypher