DEV Community: Denis MacPherson

ChaosCypher v0.4.3: When a Clean Restart Lost More Work Than a Crash

Denis MacPherson — Tue, 22 Sep 2026 20:35:42 +0000

v0.4.3 adds nothing new. No new features, no breaking API changes, no schema migrations: 54 commits since v0.4.2, 23 of them changing shipped code. Most of them are in the queue and the extraction pipeline, and most of those share one trait. The work was lost, counted twice, or quietly downgraded, and nothing reported it.

A clean restart was worse than `kill -9`

Stopping a worker gracefully destroyed work that a hard kill would have recovered.

On shutdown, the drain's CancelledError arm wrote terminal status="cancelled" to the tasks it interrupted. That is exactly the status the requeue script and retry_task refuse to resurrect. So the recovery contract ran backwards: a SIGKILL left the task recoverable, and a polite restart finished it off.

It was one of four queue-recovery defects in this release that each looked like a working system:

Crash recovery spent two units of the retry budget per cycle. requeue_atomic.lua bumped attempts on top of the worker's own claim-time increment, so a max_tries=5 task with retry_on_crash=True got 3 dispatches, not 5, and then went terminal reporting "crashed after 5 attempts". The affected operations are the four crash-retryable ones with no owning resource: execute_workflow, execute_step, export_graph, export_by_sources. For those, the queue budget is the only recovery layer.
A lapsed heartbeat could never be refreshed. refresh_heartbeat issued an EXPIRE, which is a silent no-op on a key that has already expired. A 31-second stall condemned a multi-hour task as abandoned, and the reconciler requeued it for duplicate execution. It is now a SET with EX.
A re-delivered chunk task could finish the job one chunk early. The short-circuit that made re-running an extraction chunk safe only caught a completed row. A row still in running went straight through, so two deliveries produced two terminal writes and two job-counter bumps. Both transitions are now guarded updates that return a rowcount, and the handler bails out before the billable LLM call when it loses the claim.

In plain English: if you restart workers, or your machine stalls for half a minute, queued work now comes back the number of times it was promised to, and only once.

Graph edges that explained themselves by thinking out loud

Every relationship ChaosCypher extracts carries a justification: the evidence the model gave for linking two entities. You can inspect it, and it is part of why the graph is checkable rather than taken on trust. That only works if the justification is evidence.

Some weren't. One parent_of edge at confidence 0.9 explained itself with "I will link 9 to 3 via interacts_with". That is the model deliberating, stored as if it were the reason the edge exists. Justifications that carry reasoning markers are now blanked, and the rest are trimmed to two sentences on a sentence or word boundary, so no fragment survives.

In plain English: when you read why an edge exists, you now get a reason or nothing, not the model's scratchpad.

One MCP sandbox bypass, and a read-only mode that wasn't

MCP add_document skipped its sandbox containment whenever content was supplied. The tool schema says that supplying content means the file is never loaded. Nothing implemented that: file_path was read from disk on every path and reached the loader uncontained. A caller supplying both got an arbitrary-file read that reached /data/credentials.json. The containment and dotfile guards are now unconditional, in the server and in the processor's mirror of it.

chaoscypher mcp honoured a read-only install only when you typed the flag. The guard read the --mode flag rather than the effective setting, so the documented Claude Desktop invocation against a read-configured install still handed the external client the destructive apply_upgrade tool. The guard now reads the configured mode, and an explicit --mode write still overrides it.

Also closed: the API-key selector and bearer-token throttle defects found in a post-merge sweep; decode_session rejects non-ASCII input; summarize gets the untrusted-document fence its five sibling handlers already had; and CHAOSCYPHER_ALLOW_USER_PLUGINS=0, the documented Docker kill switch for user plugins, now actually reaches all five plugin-loading services. Before, no service declared it, so the loader always saw its default.

In plain English: if you point an AI assistant at ChaosCypher over MCP, it can no longer read files outside the sandbox, and "read-only" means read-only however you launch it.

Settings that silently did nothing

Every configured trigger filter matched everything. list_triggers deferred the filters and actions columns the dispatcher reads, so a trigger scoped to one source fired on every source. The .ccx exporter and GET /workflows/{id}/triggers were losing the same keys.
A rejected workers.yaml value collapsed the Operations queue from 8 concurrent slots to 1, because a bad value fell through to a boolean instead of the worker type's default.
The commit-time orphan filter ignored the source's own settings and always dropped orphans, even for minimal or unfiltered sources and rows with protect_orphans=True.
The queue reset promised cancellation and delivered dead letters. It now runs cancel_all_tasks first, so running tasks stop instead of landing in the dead-letter set.

The API stops freezing on Cancel All

POST /queue/cancel-all ran a synchronous connect → SELECT → commit → disconnect on the event loop, once per task. After a large import that is thousands of cycles inside one request with no awaits, and Cortex served nothing else in the meantime. All four call sites now go through a worker thread. The same treatment went to per-chunk progress writes on the worker. A commit that lost the SQLite writer lock could park the loop for up to 60 seconds and starve the heartbeat refresher, which is how live tasks started to look abandoned.

The vision-page fan-out went from up to 4000 sequential round trips (bounded by vision_max_pages, default 2000) to 2. The changelog has the rest of the round: projections and unprojected reads fixed across the stuck-source sweep, step progress, queue stats and chaoscypher source search. These describe what the code was doing, not a benchmark.

Upgrading

No migrations apply. Drain the queue before you swap the image: stop new submissions and wait for /api/v1/queue/stats to report 0 pending on all queues. Don't run mixed versions against the same queue.

docker pull ghcr.io/chaoscypherinc/chaoscypher:0.4.3

Or, if you run the Python packages directly:

pip install -U chaoscypher-core chaoscypher-cortex chaoscypher-neuron chaoscypher-cli

Starting fresh:

docker run -d --name chaoscypher \
  -p 80:80 \
  -p 443:443 \
  -v chaoscypher-data:/data \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/chaoscypherinc/chaoscypher:latest

Next steps

If you use MCP with a read-configured install, upgrade first. That is the fix with the widest reach.
If you set trigger filters, check your automations after upgrading. Triggers you scoped to one source now fire only on that source.

Full details in the changelog. ChaosCypher is AGPL-3.0 and local-first: the graph, the chat, and the import and export paths all run on your own machine. Repo: https://github.com/chaoscypherinc/chaoscypher

Automate Your Knowledge Pipeline: Triggers, Workflows, and AI Tools

Denis MacPherson — Fri, 18 Sep 2026 14:46:49 +0000

Most knowledge management tools treat you like a filing clerk. Upload a document, wait for extraction, manually review entities, fix errors, tag things, connect things. Then do it all again for the next document. And the next. And the next fifty.

This is fine when you have ten documents. It falls apart at a hundred. It becomes genuinely painful at a thousand. The bottleneck is never the AI -- it's the human loop. Every document requires your attention, your judgment calls, your clicks. The extraction might take thirty seconds. Your review and cleanup take ten minutes.

Chaos Cypher's workflow engine exists to close that gap. You define a processing pipeline once -- what to extract, how to validate it, where to send notifications -- and every new document flows through it automatically. No babysitting. No repetitive clicking. You set the rules, the system follows them.

This isn't a cron job bolted onto the side. It's a proper workflow engine with event-driven triggers, conditional branching, step-to-step data passing, and composable AI tools. Let me walk you through how it works.

A Concrete Workflow: Auto-Processing Research Papers

Abstractions are boring. Let's look at a real workflow you might build: automatically processing research papers as they're uploaded.

Here's the pipeline:

Trigger: Extraction commits new entities to the graph (node.create event fires) -- the signal that a freshly uploaded paper has finished processing. (You can also invoke the workflow manually or through the API.)

Step 1 -- AI Prompt: Summarize the document in three sentences. The ai.prompt tool sends the document text to your configured LLM with instructions to produce a concise summary. If the document is long, it automatically chunks the text and processes sections in parallel, then merges the results.

Step 2 -- AI Extract JSON: Pull out structured metadata. Authors, publication date, journal name, key findings, methodology type. The ai.extract_json tool takes the document text and a JSON schema defining exactly what you want, then returns validated structured data. It retries if the extraction doesn't match the schema.

Step 3 -- Conditional: Check if this is a clinical study. The logic.conditional tool evaluates whether {{steps.extract_metadata.extracted_data.methodology_type}} equals "clinical_trial". If true, the workflow branches to run additional medical-domain extraction. If false, it skips ahead.

Step 4 -- HTTP Request: Post a notification to a Slack webhook with the summary from Step 1 and the metadata from Step 2. The http.request tool sends a POST to your webhook URL with a JSON body containing {{steps.summarize.result}} and {{steps.extract_metadata.extracted_data}}.

Notice the {{steps.summarize.result}} syntax. That's the interpolation engine at work. Every step's output is keyed by the step's ID -- reference it as {{steps.<step_id>.<field>}} from any later step. The IDs here (summarize, extract_metadata) are placeholders: IDs are instance-specific, so check a step's actual ID in the builder (or via GET /api/v1/workflows/{id}) before wiring references -- and re-check after importing a workflow, because the export format does not carry step IDs, so import assigns fresh ones and cross-step references in configurations must be updated. You can reference {{inputs.<field>}} for the data the workflow was invoked with, {{steps.extract_metadata.extracted_data.authors}} for a nested field from a previous step, or even {{steps.check_clinical.branch_taken}} to see which conditional path was followed. The interpolation preserves types too -- if a previous step returned a number, you get a number, not the string "42".

This entire pipeline runs without human intervention. Upload a PDF, walk away, come back to a summarized, metadata-tagged, conditionally-processed document with a Slack notification waiting for you.

Under the Hood: 10 Built-In Tools

The workflow engine ships with ten built-in tools organized into five categories. Each tool has a defined input schema and output schema, so the system validates your configuration before anything runs.

Category	Tools	What They Do
AI	`ai.prompt`, `ai.extract_json`, `ai.vector_search`, `ai.generate_embedding`	LLM interactions with chunking support, structured JSON extraction with schema validation and retries, semantic search across your knowledge graph, vector embedding generation for entities
Data	`data.extract`, `data.merge`	Pull values from nested objects using dot-notation paths (`user.addresses.0.city`), merge multiple dictionaries with shallow or deep strategies
Logic	`logic.conditional`, `logic.loop`	If/then branching with safe expression evaluation, iterate over collections with configurable limits
HTTP	`http.request`	External API calls with all HTTP methods, bearer/basic auth, configurable timeouts, and SSRF protection that blocks localhost access
Templates	`templates.list`	Query your knowledge graph schema to discover available node templates

Three details that make these reliable enough to run unattended: ai.prompt automatically chunks documents that exceed the model's context window, processes the chunks in parallel, and merges the results; ai.extract_json validates the LLM's output against your JSON schema and retries on mismatch; and http.request validates URLs before sending (https/http only, localhost blocked against SSRF) while supporting all standard methods and auth schemes. ai.vector_search is the interesting one for graph-aware pipelines -- workflows can semantically query existing knowledge, enabling steps like "find entities similar to what we just extracted and check for duplicates."

How Triggers Work

Triggers are the entry point for automated workflows. They listen for events in the system and fire workflows when conditions are met.

Event sources define what happened. Two are live today: node.create (a new node was committed to the graph) and edge.create (a new relationship was committed) -- both fire on the extraction-commit path. Other event sources, like file-upload and import-complete, are selectable in the builder but not yet emitted by the engine. The system ships with a built-in auto-embedding trigger -- nodes created when an extraction commits to the graph automatically get vector embeddings generated.

Filters let you narrow the scope, using exact key-value matching against the event data. A trigger on node.create with a filter {"entity_type": "node"} fires only for node events. The node.create payload currently carries entity_type and entity_id, so filters on other keys (like a template id) can never match -- the engine logs a warning when a trigger is wired to a structurally impossible filter.

Statistics tracking gives you visibility. Every trigger execution records success/failure status, execution time, and error messages. You can see your success rate, average execution time, and recent execution history -- useful for debugging workflows that occasionally fail.

The "Expose as AI Tool" Feature

Here's where things get composable. Any workflow can be exposed as a callable AI tool by setting expose_as_ai_tool: true and defining input/output schemas. Once exposed, that workflow appears alongside the built-in tools and can be used as a step in other workflows.

Think about what this enables. You build a workflow that extracts and validates medical terminology. You expose it as a tool. Now your "process research papers" workflow can call it as Step 3 instead of hardcoding medical-domain logic. You have a workflow that enriches person entities by cross-referencing external APIs? Expose it, and any other workflow can use it.

Workflows calling workflows. Each one focused on a single job, composed together into pipelines of arbitrary complexity. The step type workflow (alongside system_tool and user_tool) tells the engine to execute another workflow as a step, passing inputs and receiving outputs just like any other tool.

Workflow Portability

Workflows export to a version-stamped, self-describing JSON file -- the definition, all steps, all configurations, no hidden state, no implicit database IDs. Import validates the version and resolves every referenced tool against the target instance's registry before creating anything, so a missing tool produces a clear error instead of a broken workflow. Export from your laptop, import on a server, share with a colleague.

Try It Yourself

The fastest way to see the workflow engine in action is to look at the export format. Here's a minimal workflow that summarizes documents on upload:

{
  "version": "1.0",
  "workflow": {
    "name": "Summarize on Upload",
    "description": "Auto-summarize new documents when uploaded",
    "input_schema": {
      "type": "object",
      "properties": {
        "document_text": {
          "type": "string",
          "description": "The document content to summarize"
        }
      },
      "required": ["document_text"]
    },
    "output_schema": {
      "type": "object",
      "properties": {
        "summary": {
          "type": "string",
          "description": "Three-point summary of the document"
        }
      }
    }
  },
  "steps": [
    {
      "step_number": 1,
      "name": "Summarize Document",
      "tool_type": "system_tool",
      "tool_id": "ai.prompt",
      "configuration": {
        "prompt": "Summarize this document in 3 key points:\n\n{{inputs.document_text}}",
        "output_format": "text"
      }
    }
  ]
}

This is everything the system needs. The version field ensures forward compatibility. The input_schema and output_schema define the contract. The steps array contains the pipeline.

Each step specifies its tool_type (system_tool, user_tool, or workflow), a tool_id that references a registered tool, and a configuration object whose shape matches the tool's input schema. The {{inputs.document_text}} template variable gets resolved at execution time with the inputs the workflow was invoked with.

To wire it up, create a trigger with the event source (like node.create), link it to your workflow, and optionally add filters. From there the engine scales with you: steps can declare depends_on, continue_on_error, max_retries (per-step, falling back to the workflow-level default), and timeout_seconds; imports handle name conflicts (fail/skip/rename) and can land inactive for a test run first. The workflows API reference documents the step fields and import options.

One more thing worth knowing before you trust a pipeline to run unattended: every run produces an execution record -- status, inputs, outputs, per-step timing, and which step failed and why. When a workflow fails at 3am, you don't guess. You look at the execution detail, see that Step 3 timed out after 120 seconds waiting for the LLM, and adjust accordingly.

What's Next

The workflow engine is designed to grow. The tool system uses a plugin architecture -- the same pattern that powers Chaos Cypher's loader plugins, domain plugins, and LLM providers. Custom tool plugins in Python shipped: drop a *_plugin.py file in data/plugins/tools/ implementing tool_id, category, name, description, input_schema, and an execute method, and it auto-registers on startup (user plugins even override built-ins with the same tool_id). See the tool plugins guide for the full interface.

More trigger event sources are coming as the platform grows. Scheduling (run a workflow every Tuesday at 9am) and webhook triggers (fire a workflow from an external system) are natural extensions of the existing event-driven architecture.

If you've built an interesting automation workflow -- whether it's a multi-step research pipeline, a quality assurance checker, or an integration with external tools -- I'd genuinely like to hear about it. The export format makes sharing straightforward: export your workflow, share the JSON, and someone else can import it and adapt it to their use case. That's the whole point of portability.

For the full API reference and detailed configuration options, check out the workflow documentation. The built-in system workflows (like auto-embedding on node create) are also good starting points -- export them and study the step configurations to see how the engine's own automation is wired together.

From Obsidian Vault to AI-Powered Knowledge Graph in Minutes

Denis MacPherson — Fri, 18 Sep 2026 14:46:28 +0000

Open your Obsidian graph view and look at it honestly: every line on that screen is a link you typed by hand. The person mentioned in forty notes but never [[linked]], the concept that appears under three different names, the two projects connected by a decision you wrote down once and forgot -- none of it shows up. Your vault knows more than its graph does.

Chaos Cypher adds the missing layer. It reads the Markdown itself, extracts entities and relationships with an LLM, and makes the whole vault searchable and chat-ready -- without requiring a single connection to be pre-linked. Your manual wiki-links stay exactly what they are: deliberate structure. The AI-discovered layer sits alongside them, surfacing the connections you never typed.

The Workflow

No plugin required -- the archive loader does the work:

Zip the notes you want to analyze. Start with a focused folder of at least ~10 notes rather than the entire vault, and leave out private notes you don't want processed.
Upload the ZIP (or TAR.GZ) as a source. The archive loader recognizes Markdown-heavy projects, strips frontmatter, preserves heading structure, and sends each note through the same pipeline used for PDFs and web pages. Smaller archives are processed by the generic loader, which keeps frontmatter intact -- bundle at least ten notes to engage the Markdown handler's frontmatter stripping.

Confirm the detected domain. Indexing (chunking + local embeddings) takes seconds and immediately enables semantic search and RAG chat. After indexing, Chaos Cypher analyzes your notes and proposes an extraction domain -- review and confirm (or override) it in the dialog, and extraction starts.
Wait for extraction. Extraction builds the graph: typed entities and relationships pulled from the prose itself.
Explore what the AI found. People mentioned across multiple notes, concepts that appear under different headings, projects connected by shared decisions, recurring themes that were never explicitly linked -- now visible, navigable, and traceable back to the exact notes they came from.

Ask questions. "What decisions led to the current architecture?" or "Where do my notes on attention mechanisms and retrieval overlap?" -- answered with citations back to your own notes, not the internet.

Scripting the Import

The steps above walk through the UI, but the same pipeline is reachable without opening a browser.

For developers: POST /api/v1/sources takes a multipart file upload and runs it through the identical loader dispatch the UI uses -- point it at your vault archive and it works the same way. POST /api/v1/sources/batch accepts a list of files if you're importing several exports at once, which is handy wired into a nightly vault-backup job: export, POST the archive, let indexing and extraction pick it up. In plain English: anything the "Upload" button does, a script can do too, over HTTP.

For CLI users: the same import is one command:

chaoscypher source add my-vault.zip

source add isn't Obsidian-specific -- it's the general-purpose source-loading command, and it routes to the archive loader the same way the web UI does whenever it sees a .zip or .tar.gz. There's no separate "import vault" subcommand, because none is needed. In plain English: if you'd rather stay in the terminal, one command gets your vault into the graph.

A Few Tips for a Good First Pass

Keep filenames and headings descriptive -- they become context the extractor uses.
If your vault has a specialized vocabulary, review the extracted graph and then tune domains or templates to match it.
Everything runs locally if you pair it with Ollama -- your journal never leaves your machine.

This workflow shines wherever the value isn't only in the notes but in the relationships between them: research notes, project journals, meeting archives, long-running personal knowledge bases.

What's Next

A first-class Obsidian importer -- reading vaults directly, preserving wiki-links as graph edges alongside the AI-discovered ones -- is on the roadmap. If that's your use case, tell us how you'd want it to work; vault structures vary wildly and real examples shape the design.

Extract Smarter: How Domain-Aware AI Builds Better Knowledge Graphs

Denis MacPherson — Fri, 11 Sep 2026 14:47:08 +0000

Most AI extraction tools treat every document the same way. Upload a medical paper or a legal contract and you get the same generic entity types, the same vague relationships, the same disappointing graph. Chaos Cypher takes a different approach: it detects what kind of document you uploaded and adapts its entire extraction pipeline to match.

The Problem with Generic Extraction

Here's a sentence you might find in a clinical document:

Patient with hypertension started on lisinopril 10mg daily. The ACE inhibitor is contraindicated with potassium supplements. Side effects include dry cough and dizziness.

A generic extraction pipeline -- the kind most tools use -- will pull out a handful of entities and connect them with whatever relationship labels the LLM feels like inventing. You might get "Lisinopril" typed as an Item, "Hypertension" as a Concept, and "Dry Cough" as another Concept. The relationships between them? Probably related_to and influences. Maybe associated_with if you are lucky.

This is the "garbage in, garbage out" of knowledge graphs. It's not that the AI failed to read the text. It read it fine. The problem is that nobody told it what to look for, what types are valid, or what the relationships between those types should mean.

The graph you get is technically correct and practically useless. You cannot query "which drugs treat hypertension" because the system does not know what a Drug is. You cannot find contraindications because related_to could mean anything. Every edge in the graph carries the same semantic weight as a shrug.

Now run the same sentence through Chaos Cypher with the medical domain active:

Lisinopril becomes a Drug with dosage form and mechanism of action as properties
Hypertension becomes a Condition
Dry Cough and Dizziness become Side Effects
Potassium Supplements gets recognized as a Drug (because supplements have drug interactions too)

The relationships are just as precise: treats, contraindicated_with, produces_side_effect. Each one is typed, directional, and constrained. A treats edge must start from a therapeutic agent (Drug, Treatment, Procedure, Protocol, or Guideline) and land on a clinical target (Condition, Symptom, Side Effect, Pathogen, or Patient Population). A Side Effect can never treat a Drug. The LLM isn't guessing -- it's following a schema.

That's what domain-aware extraction does. It turns a language model from a general-purpose pattern matcher into a domain specialist.

How It Works: Upload to Knowledge Graph

The workflow is straightforward. You upload a document. Chaos Cypher figures out what domain it belongs to, loads the right extraction rules, proposes the domain for your confirmation, then runs the pipeline. You don't need to configure anything upfront -- though you can override the detected domain if you want.

:::info Update (June 2026)
Since the domain-confirmation gate shipped, auto-detection proposes a domain and parks the source awaiting your one-click confirmation before the (potentially hour-long) extraction runs. You can accept, override the proposed domain, or pass auto_confirm at upload to restore the fully hands-off behavior described below.
:::

Here's what happens behind the scenes:

Detection -- Chaos Cypher samples up to ~12,000 characters drawn from the beginning and the middle of your document and scores it against all registered domains simultaneously. Each domain has weighted keyword groups, regex patterns, and file type signals. The highest-scoring domain wins.
Guidance injection -- The winning domain's extraction rules get injected into the LLM prompt. This includes entity type definitions, relationship constraints, exclusion rules (what not to extract), and worked examples of correct extractions.
Strict type enforcement -- The LLM is instructed to only use entity types from the domain's template list. After extraction, a code-level filter drops any entity whose type does not match a known template. No hallucinated types survive.
Relationship validation -- Each relationship is checked against source/target type constraints. A treats relationship must flow from a therapeutic agent like a Drug, Treatment, or Procedure to a clinical target like a Condition or Symptom. Anything else gets rejected.
Quality scoring -- Extracted entities and relationships are scored by domain relevance. Domain-specific types like Drug and Condition score higher than generic fallbacks. This surfaces the most valuable parts of your graph.

Chaos Cypher ships with 19 built-in domains, each tuned for a different category of document. A sample:

Domain	Typical Entity Types	Best For
Medical	Drug, Condition, Symptom, Procedure, Side Effect	Clinical documents, pharmaceutical literature
Technical	Module, Class, Function, Endpoint, Design Pattern	API docs, codebases, technical specifications
Legal	Statute, Case, Party, Obligation, Legal Principle	Contracts, court opinions, regulatory filings
Financial	Company, Financial Instrument, Market Event, Regulation	Earnings reports, market analysis, SEC filings
Scientific	Hypothesis, Method, Finding, Dataset, Organism	Research papers, experiments, academic publications
Cybersecurity	Threat Actor, Vulnerability, Malware, Attack Technique	Threat intel, incident reports, CVE research
Generic	Person, Organization, Event, Concept, Location	General-purpose fallback for any content

The rest cover historical, literary, theological, investigative, political, design, intelligence, news, educational, philosophical, biographical, and reference material — the full list with every entity type is in the docs.

Every domain uses strict entity type enforcement by default. The medical domain defines 20 entity types. The technical domain has 18. These aren't suggestions -- they're the only types the LLM is allowed to produce. That constraint is what separates a clean, queryable graph from a noisy soup of ad-hoc labels.

Under the Hood: Domain Detection and Extraction Quality

How Detection Works

Domain detection runs a scoring algorithm across all registered domains simultaneously. Each domain defines its detection rules in a JSON-LD config file with three signal types:

Weighted keyword groups. The medical domain has six keyword groups: clinical_core (weight 1.2), pharmaceutical (weight 1.0), diagnostic (weight 0.9), anatomy (weight 0.8), procedures (weight 0.9), and clinical_terms (weight 0.8). Each keyword match boosts the confidence score by per_keyword_boost * weight. A document full of "diagnosis", "treatment", and "symptoms" racks up points fast in the clinical_core group, while scattered mentions of "cardiac" and "pulmonary" add smaller anatomy-weighted boosts.

Regex patterns. Keywords catch common terms, but patterns catch domain-specific notation. The medical domain matches dosage expressions like \d+\s*(mg|mcg|ml), ICD codes like ICD-10:J45, and prescription abbreviations like b.i.d. and p.r.n.. Each pattern match carries its own weight -- dosage notation at 1.4x, ICD codes at 1.5x. A single ICD code in a document is a strong medical signal.

File and document type signals. File extensions (.py for technical) and document type metadata (medical_document, openapi) provide additional boosts.

The final confidence score is compared against a per-domain minimum threshold. Medical requires 0.4 minimum confidence. The generic domain has a threshold of 0.0 -- it always matches as a fallback, but with the lowest possible score (0.1), so any specialized domain that passes both its own minimum threshold and the registry's absolute confidence floor of 1.0 will win. The floor exists so weak, ambiguous matches fall back to the broad generic schema instead of locking in a wrong domain.

How Domains Shape Extraction Quality

Detection picks the right domain. But the real value is in what happens next -- the selected domain controls the extraction pipeline at five points:

Entity guidance tells the LLM what to extract and what to skip. The medical domain says "include dosage information as properties on drug entities" and explicitly excludes standalone dosage numbers, study references ("Figure 1"), and administrative codes.
Strict type enforcement gives the LLM a closed list of valid types -- the medical domain allows exactly 20. Anything outside that list gets dropped in post-processing. No more "Medical Concept" or "Health Thing" cluttering your graph.
Relationship constraints validate source and target: treats must flow from a therapeutic agent like a Drug, Treatment, or Procedure to a clinical target like a Condition or Symptom. A Symptom that "treats" a Drug fails validation. This catches the most common extraction error -- reversed or nonsensical edges.
Compatibility groups handle deduplication: "Hypertension" extracted as a Condition in one chunk and a vaguer type in another can be merged because both belong to the clinical group -- no duplicates, no lost type precision.
Property type mapping rescues mistyped entities: "Severity" extracted as a standalone node gets absorbed into the right Condition as a severity property instead of floating as an orphan.

In plain English: the domain is a contract. The LLM does the reading, but the domain decides what counts as a valid entity, a valid relationship, and a valid graph -- and everything that breaks the contract is filtered out before it reaches you.

Try It Yourself

Every built-in domain is just a JSON-LD file. No Python, no compilation, no framework code. If you need a domain for your field that doesn't exist yet, you can create one in about 20 minutes: drop a file in data/plugins/domains/, define your detection signals and your vocabulary, restart, done.

Say you want a startup domain for pitch decks and funding announcements. The skeleton is just three parts -- how to recognize the content, what entities exist, and how they're allowed to connect:

{
  "@type": "ExtractionDomain",
  "name": "startup",
  "strict_entity_types": true,

  "detection": {
    "confidence": { "base_score": 0.2, "per_keyword_boost": 0.12 },
    "keywords": {
      "funding": {
        "terms": ["series A", "series B", "seed round", "venture capital",
                  "valuation", "term sheet", "cap table", "pre-money"],
        "weight": 1.3
      }
    },
    "patterns": [{ "regex": "\\$\\d+[MBK]\\s+(seed|series|round)", "weight": 1.5 }]
  },

  "templates": {
    "node_templates": [
      { "id": "startup_company", "name": "Company" },
      { "id": "startup_round", "name": "Funding Round" }
    ],
    "edge_templates": [
      { "id": "startup_raised", "name": "raised",
        "source_types": ["Company"], "target_types": ["Funding Round"] }
    ]
  }
}

Detection has an absolute confidence floor of 1.0 -- a domain that scores below it loses to the generic fallback, so give a new domain enough keyword surface to clear it, or force the domain at upload while you tune detection. (The skeleton above clears it: eight keywords at a 0.12 boost and 1.3 weight put a keyword-rich article at roughly 1.45 before patterns even fire.)

Upload a TechCrunch article and watch the detection engine propose it at the confirmation step: confirm, and Companies and Funding Rounds appear in your graph, connected only by the relationships you allowed.

The full walkthrough -- weighted keyword groups, named-referent flags, entity properties, inverse relationships, and a complete worked example -- lives in the docs: start with Custom Domains, then Building Extraction Domains for the whole schema, including normalization keywords, compatibility groups, property absorption, and extraction-density tuning. The built-in medical domain is the most comprehensive reference -- study it when you want the full picture.

What's Next

We're planning more specialized domains -- supply chain, environmental science, and music theory are on the shortlist. But the real potential is in what users build. Every field has its own vocabulary, its own entity types, its own relationship patterns. A materials scientist cares about Crystal Structure, Synthesis Method, and Property. A genealogist needs Person, Family, Vital Record, and Census Entry. A cybersecurity analyst -- who already has a built-in domain -- might want to fork it and add types specific to their organization's threat model.

If you build a domain for your field, share it. A JSON-LD file is small, portable, and easy to review. Drop it in data/plugins/domains/ and it works. No pull request required to use it, but we would love to include community domains in the built-in set for others to benefit from.

Domains work identically whether you're running locally with Ollama or with a cloud provider. Start with the skeleton above, test it on your documents, and iterate from there.

Give Any AI Assistant Direct Access to Your Knowledge Graph with MCP

Denis MacPherson — Fri, 11 Sep 2026 14:45:42 +0000

Your knowledge graph is stuck in a browser tab. You built something valuable -- a map of entities, relationships, and source documents that represents real understanding of a domain. But the moment you switch to Claude to write a report, or open Cursor to write code, or ask ChatGPT to help with analysis, that knowledge graph might as well not exist. You're back to copying text, pasting context, and manually cross-referencing. Two tools that should be working together are stuck in separate worlds.

Chaos Cypher now speaks MCP, which means any AI assistant that supports the protocol -- Claude Desktop, Claude Code, Cursor, Windsurf, and a growing list of others -- can directly query, search, traverse, and even write to your knowledge graph. No copy-paste. No context switching. Just ask.

This post walks through what that actually looks like, what's under the hood, and how to set it up in about two minutes.

What Is MCP, and Why Should You Care?

MCP stands for Model Context Protocol. Anthropic released it as an open standard, and the simplest analogy is USB-C for AI tools. Before USB-C, every device had its own charger, its own cable, its own connector. MCP does the same thing for AI integrations: it defines one protocol that any AI host can use to talk to any tool server.

Instead of building a custom plugin for Claude, another for ChatGPT, another for Cursor, and another for every new AI tool that launches next month, you build one MCP server. Every compatible AI tool can use it immediately.

The adoption has been fast. Claude Desktop, Claude Code, Cursor, Windsurf, Cline, and Continue all support MCP today. The protocol handles tool discovery (the AI asks "what can you do?"), tool invocation (the AI calls a function with parameters), and result streaming. From the AI's perspective, your knowledge graph becomes just another set of capabilities it can use to answer questions.

From your perspective, it means you stop being the middleman between your data and your AI.

What This Actually Looks Like

The best way to understand MCP is to see the before and after.

Before MCP: You have a knowledge graph with 200 entities extracted from research papers on gene therapy. You're writing a literature review in Claude. To reference your graph, you open Chaos Cypher in another tab, run a search, copy the results, paste them into Claude, ask your question, realize you need more context, go back to the graph, find related entities, copy those too, paste again. Repeat until frustrated.

After MCP: You tell Claude: "Search my knowledge graph for all entities related to CRISPR and find the shortest path to gene therapy applications." Claude calls graphrag_search to find relevant entities and document passages, then calls find_shortest_path to trace the relationship chain. You get a grounded answer with specific entities and relationships from your own research, in one turn.

Here are three scenarios that show the range of what's possible.

Scenario 1: Research -- Connecting the Dots

You've been building a knowledge graph from papers on quantum computing and machine learning. You're deep in a writing session in Claude Desktop and want to understand where these two fields intersect in your collected research.

You ask: "What are the connections between quantum computing and machine learning in my research? Show me the key entities and how they're related."

Claude calls search_nodes to find nodes matching both topics, then get_node_context to pull the immediate neighborhood of the most central ones, including the edges that connect them and the source document chunks that support each relationship. You get back a structured map of how your research connects these fields -- not a generic internet answer, but one grounded in the specific papers you've indexed.

Scenario 2: Coding -- Your Project's Knowledge Base in Your Editor

You're in Cursor, working on a codebase that has an associated knowledge graph mapping its architecture -- services, APIs, data flows, dependencies. You need to understand how the authentication service connects to the billing pipeline.

You ask: "Traverse from the Authentication Service node to anything related to billing. What's the path?"

Cursor calls resolve_node to find the canonical node for "Authentication Service" (even if you didn't remember the exact label), then traverse_path to walk the graph two hops out, filtered to the relevant edge types. You see the chain: Authentication Service -> User Session -> Subscription Manager -> Billing Pipeline. Without leaving your editor.

Scenario 3: Writing -- Summarize With Citations

You're drafting a report and need to summarize everything in your knowledge graph about a specific topic, with citations back to the original source documents.

You ask: "Summarize all my sources related to climate policy in the European Union. Include which documents each claim comes from."

Claude calls get_summary_context to retrieve and cluster document chunks relevant to the query. Because this tool returns the raw chunks with their source metadata rather than making an LLM call, Claude itself does the summarization -- giving you a synthesis grounded in your documents, with each claim traced back to a specific source.

Under the Hood: 31 Tools, 7 Categories

Chaos Cypher exposes 31 tools through MCP, organized into seven categories: GraphRAG search (the flagship -- Personalized PageRank fused with hybrid vector/keyword retrieval), nodes, edges, templates, analytics (shortest paths, similarity, community detection, multi-hop traversal), documents (upload, status, summarization context), and client-driven extraction -- where the AI assistant reads chunks, extracts entities itself, and submits them back, no server LLM required. The full tool reference is in the docs.

The design principle: read operations are always safe and always available, write operations are opt-in. 19 tools are read-only; 12 require write mode to be explicitly enabled by a single setting. If you're not comfortable with an AI modifying your graph, leave it in read mode -- the AI can still search, traverse, and analyze everything.

Two transport modes: The MCP server runs in two ways depending on your setup:

stdio -- For desktop AI tools like Claude Desktop and Cursor. The CLI starts a server that communicates over standard input/output. No network involved.
Streamable HTTP -- For the Docker stack. The Cortex API exposes MCP at /api/v1/mcp using the Streamable HTTP transport, so MCP clients on your network can connect after authenticating.

Both transports expose the same 31 tools with the same behavior. The only difference is how they're connected.

Try It Yourself

If you have the Chaos Cypher CLI installed, connecting Claude Code is one command:

claude mcp add chaoscypher -- chaoscypher mcp

For Claude Desktop, add the equivalent two-line entry to claude_desktop_config.json:

{
  "mcpServers": {
    "chaoscypher": { "command": "chaoscypher", "args": ["mcp"] }
  }
}

That's genuinely it -- restart the client and Chaos Cypher's tools appear automatically. Cursor and other stdio clients use the same two-line config, and if you run the Docker stack, the MCP endpoint is already live at http://localhost/api/v1/mcp for any Streamable-HTTP client that signs in with your Chaos Cypher credentials -- it sits behind the same edge authentication as the rest of the API (or http://localhost:8080/api/v1/mcp if you run the multi-container dev stack, which publishes Cortex's port directly). Per-client walkthroughs are in the MCP setup docs.

By default the server runs in read-only mode; flip mcp.mode: write in settings.yaml to enable the 12 write tools, and use chaoscypher mcp --database my-research to point at a specific database. The configuration reference covers the rest, including auto_extract for documents uploaded via MCP.

Your Data Stays Local

This is worth stating explicitly: MCP doesn't send your knowledge graph data to any external service. The protocol is a local communication channel between the AI tool running on your machine and the Chaos Cypher server running on your machine (or your network, if you use Docker). When Claude calls graphrag_search, the query goes from Claude to your local MCP server, your server searches your local database, and the results go back to Claude. Your documents, entities, and relationships never leave your infrastructure.

The AI model itself runs wherever it runs -- that's between you and your provider. But the knowledge graph data stays entirely under your control. If you pair Chaos Cypher with a local model via Ollama, the entire pipeline is air-gapped. See our local AI setup guide for the full walkthrough.

What's Next

MCP support is the foundation for a broader vision: your knowledge graph as a persistent layer that any tool in your workflow can tap into. Here's what's on the roadmap:

Prompt templates -- Pre-built MCP prompts for common patterns like "summarize this topic with citations" or "find contradictions in my sources," so you don't have to craft the right question every time.
Resource exposure -- Making graph nodes and documents available as MCP resources, so AI tools can browse your knowledge graph like a file system.
Multi-database switching -- Seamlessly switch between knowledge graphs within a single MCP session.

The flagship graphrag_search tool deserves its own explanation -- it's doing a lot more than keyword lookup. Read how GraphRAG works for the full deep-dive on the retrieval pipeline.

The MCP server ships with Chaos Cypher today. If you're already running it, you have it -- just configure your AI tool and go.

Documentation: Full MCP setup guide and tool reference in the docs
Source: The MCP implementation lives in the chaoscypher_core.mcp package.
Issues: Found a bug or have a feature request? Open an issue or start a discussion

The gap between "having a knowledge graph" and "using a knowledge graph" has always been the friction of switching contexts. MCP closes that gap. Your knowledge graph is no longer a destination you visit -- it's a capability that follows you into whatever tool you're already working in.

Why Your RAG Chat is Missing Half the Answers (And How GraphRAG Fixes It)

Denis MacPherson — Fri, 11 Sep 2026 14:45:41 +0000

You upload four research papers to your RAG chatbot. You ask: "How does Dr. Chen's CRISPR research connect to the gene therapy trials at Stanford?" The chatbot thinks for a moment and gives you... a paragraph about CRISPR. Generic, shallow, pulled from whichever single chunk happened to mention the word. The actual answer -- that Chen published a paper on CRISPR delivery mechanisms, which was cited by a Stanford clinical trial for retinal gene therapy, which built on a funding collaboration between both institutions -- exists across three different documents. Your chatbot never even tried to find it.

This is the multi-hop problem, and it's the silent failure mode of every vector-only RAG system. Vector search embeds your question, compares it against document chunks, and returns the closest matches by cosine similarity. It works for single-hop questions: "What is CRISPR?" or "When did the Stanford trial begin?" But the moment an answer requires connecting information across documents -- following a citation chain, tracing a person through multiple sources, linking a cause in one report to an effect in another -- vector search falls apart. It can't follow relationships. It doesn't know that entities in different documents refer to the same thing. It just sees text. In an independent benchmark, GraphRAG-style retrieval scored 53% vs 43% for reranked vector RAG on complex multi-hop reasoning (GraphRAG-Bench, ICLR 2026).

The worst part: it fails silently. No error message, no "I couldn't find a complete answer." You get a confident-sounding response that happens to be shallow or wrong.

Chaos Cypher's GraphRAG search fixes this by fusing knowledge graph traversal with vector search. When you ask a multi-hop question, it walks the graph of entities and relationships extracted from your documents, finds structurally connected information you didn't ask about, retrieves the source passages that prove those connections, and merges everything into a single ranked result set. The answer you get isn't just semantically similar text. It's the actual chain of evidence.

What Happens When You Ask a Multi-Hop Question

Let's walk through a real scenario. You have uploaded three documents into Chaos Cypher: a research paper by Dr. Sarah Chen on CRISPR delivery vectors, a Stanford clinical trial report on retinal gene therapy, and a grant proposal connecting both institutions. You type into the chat: "How does Chen's CRISPR work relate to the Stanford gene therapy trial?"

Here's what happens behind the scenes, in seven steps.

Step 1: Embed the query. Your question gets converted into a vector embedding -- the same starting point as any RAG system.

Step 2: Match seed entities. Instead of immediately searching document chunks, GraphRAG first searches the knowledge graph. It finds entities whose embeddings are closest to your query vector. In this case, it matches "Dr. Sarah Chen" (a Person node) and "CRISPR delivery vectors" (a Concept node) as high-confidence seeds -- the anchor points for graph exploration.

Step 3: Personalized PageRank. This is where it gets interesting. Standard PageRank finds globally important nodes. Personalized PageRank is different: it starts from your seed entities and performs a biased random walk through the graph. At each step, there is an 85% chance of following a relationship to a neighbor, and a 15% chance of teleporting back to a seed. Entities structurally close to your seeds get high scores, even if they were never mentioned in your query.

In our example, the algorithm discovers that "Dr. Sarah Chen" has a "published" relationship to "Lipid Nanoparticle Delivery Study," which has a "cited_by" edge pointing to "Stanford Retinal Gene Therapy Trial Phase II," which in turn has a "funded_by" connection to "NIH CRISPR Therapeutics Grant" -- a grant that also lists Chen as a co-investigator. None of these intermediate entities matched your query by text similarity. The graph surfaced them.

Step 4: Assemble graph context. The top-scoring entities from PageRank are collected along with their relationships. This produces a structured context: seed entities you asked about, related entities the graph discovered, and the relationship triples connecting them. This context gets passed to the language model alongside the document chunks, giving it the structural "map" it needs to reason about connections.

Step 5: Retrieve provenance chunks. The first of two independent retrieval paths. For each entity the graph surfaced, GraphRAG looks up which document chunks those entities were originally extracted from. Chen was extracted from page 3 of the research paper. The Stanford trial came from the clinical report abstract. The funding connection came from page 12 of the grant proposal. These "provenance chunks" contain the actual evidence for the graph relationships.

Step 6: Retrieve vector chunks. The second path runs next -- standard hybrid search (semantic + keyword) against all document chunks. It catches relevant passages that might not have generated graph entities but still contain useful context.

Step 7: Merge and rank. The two paths produce two independently ranked lists. GraphRAG merges them using Reciprocal Rank Fusion, which combines rankings without normalizing scores across systems. Chunks appearing in both lists get a combined boost. The result is a single, deduplicated, ranked list of the most relevant passages across all your documents.

Instead of a shallow answer about CRISPR, you get the full chain: Chen's delivery mechanism research led to a cited clinical application at Stanford, connected through shared funding. The chat response includes both the graph context (discovered entities and relationships) and the document passages that prove those connections.

Under the Hood (Technical Deep-Dive)

This section is for developers who want to understand the algorithms. Skip ahead to "Try It Yourself" if you just want to use it.

Personalized PageRank

Standard PageRank models a "random surfer" following links uniformly across a network. Personalized PageRank changes one thing: instead of teleporting to a random node, the surfer teleports back to seed nodes. This transforms a global importance metric into a query-specific relevance metric.

Chaos Cypher's implementation uses rustworkx's compiled power iteration with a 0.85 damping factor and runs entirely in-process -- no external graph database or service required. The seed weights come from the vector similarity scores in Step 2, so the random walk isn't just seeded on the right entities -- it's biased toward the ones most relevant to your specific question.

Reciprocal Rank Fusion

Provenance chunks have graph-connectivity scores. Vector chunks have cosine similarity scores. These aren't on the same scale, so you can't just sort by score.

RRF (Cormack, Clarke & Butt, 2009) sidesteps this by ignoring scores entirely and using only rank positions. Each chunk's RRF score is the sum of 1 / (k + rank) across all lists where it appears. The smoothing constant k (60, matching the original paper) dampens the advantage of being ranked first versus second.

The key property: chunks appearing in both lists get contributions from both, naturally boosting results validated by two independent signals. A chunk ranked 5th in provenance and 8th in vector search will often outrank one that is 1st in vector but absent from provenance. Evidence confirmed by graph structure is worth more than text similarity alone.

Graceful Degradation

Not every database has a knowledge graph. Not every query matches graph entities. GraphRAG picks its operating mode automatically:

full_graphrag -- Seeds found, PPR succeeded. Graph context + provenance chunks + vector chunks + RRF fusion.
vector_only -- Embeddings work but no graph seeds found. Standard hybrid search, no graph context.
keyword_only -- Embeddings unavailable. Pure SQLite FTS keyword search.

The system never fails -- it always returns the best results it can. The retrieval stats in each response tell you exactly what happened: mode used, seeds found, entities explored, provenance versus vector chunk counts.

In plain English: the graph finds the path between things you asked about, the vector index finds text that sounds like your question, and the fusion step trusts evidence confirmed by both more than either alone.

Try It Yourself

Here's the good news: you don't need to configure anything. GraphRAG is the default search mode behind every chat conversation in Chaos Cypher. When you type a question, the chat system prioritizes graphrag_search as its first tool for most questions. If your database has extracted entities and embeddings, you get the full pipeline. If not, it degrades gracefully to vector or keyword search.

The simplest way to see it in action:

Upload 3-4 related documents. Pick sources that share entities -- research papers from the same field, chapters from the same book, reports about the same project. The key is overlap: the documents should reference some of the same people, organizations, concepts, or events.
Wait for extraction to complete. Chaos Cypher will chunk the documents, generate embeddings (automatic), and then you can optionally run entity extraction to build the knowledge graph. (Update, May 2026: extraction now auto-detects your document's domain first and asks you to confirm it before the run starts -- you can also pre-confirm in the upload dialog.) The extraction step is what creates the graph nodes and edges that GraphRAG traverses. Without it, you still get vector-only search, which is fine -- but you miss the multi-hop connections.
Ask a question that spans documents. Don't ask something that a single document can answer. Ask about connections: "How does X relate to Y?" or "What is the link between the findings in paper A and the methodology in paper B?" This is where GraphRAG earns its keep.
Check the retrieval stats. In the chat response metadata, you'll see the retrieval mode (full_graphrag, vector_only, or keyword_only), the number of seed entities found, how many entities PageRank explored, and the breakdown of provenance versus vector chunks. This tells you exactly what the pipeline did for your query.

GraphRAG is also available as an MCP tool called graphrag_search, meaning any AI assistant that supports MCP can use it directly against your Chaos Cypher instance. See our MCP launch post for setup instructions with Claude Desktop, Cursor, and others.

The pipeline is tunable -- seed similarity threshold, PageRank top-K and damping, triple limits -- via a graphrag section in settings.yaml, though the defaults were chosen from the GraphRAG literature and testing across database sizes, and most users will never need to touch them. The full parameter reference is in the search documentation.

What's Next

GraphRAG in Chaos Cypher today handles local queries well -- questions where you have a specific starting point and want to follow connections outward. But there's a class of questions it doesn't yet handle optimally: corpus-wide questions like "What are the main themes across all my documents?" or "Summarize everything related to sustainability."

These require what the research literature calls community summaries -- pre-computed summaries of entity clusters in the graph that can answer high-level questions without traversing the entire structure at query time. That's on the roadmap.

If you're working with a use case where multi-hop retrieval matters -- legal discovery, academic research, intelligence analysis, medical literature review -- we'd love to hear about your experience. What kinds of multi-hop questions does your work require? Where does the current pipeline fall short? The best way to reach us is through the project's GitHub discussions.

For a deeper look at the architecture, see the Search documentation and the Architecture overview.

ChaosCypher v0.4.2: The Streaming Bug That Only Showed Up When the Chat Was Working

Denis MacPherson — Wed, 09 Sep 2026 00:26:29 +0000

v0.4.2 adds nothing. No new features, no breaking API changes, no schema migrations — 61 commits of fixes on top of v0.4.1. If you self-host ChaosCypher, three of them matter enough to upgrade for: live chat streaming works again, a queued task could create database files wherever it liked, and the API stopped shipping whole payloads on every poll.

The bug that hid behind a working feature

Live chat streaming was dead, and the product looked fine anyway.

The middleware that disconnects request-scoped storage adapters ran its teardown as soon as the response object existed — not when the response had finished sending. On GET /chats/{id}/events the generator's first suspension is the pub/sub subscribe, and the reconcile right after it hit an adapter that had just been disconnected. So every live stream died with STREAM_INTERNAL_ERROR before relaying a single token, whenever the chat was actually processing.

What made it survive review is what happened next: the answer still appeared when you reloaded. The worker had run, the turn was persisted, the data was correct. Only the live token-by-token experience was gone — which reads as "streaming feels broken today" rather than "a middleware is tearing down storage under an open response".

The middleware is now pure ASGI and tears down after the whole response is sent. That fixes the class, not just the symptom: any streaming endpoint or background task that touches an adapter after the response object is produced was exposed to the same teardown. It previously had no tests at all; it now has seven, two of which fail against the old implementation.

In plain English: chat answers stream live again, and the piece of plumbing that broke them is now covered by tests.

Three security fixes worth reading if you run this on a LAN

A queued task could create SQLite files at a path of its choosing. metadata.database_name on POST /api/v1/queue/tasks reached a bare Path join in get_db_path, which then mkdir'd and created a database file wherever the value pointed. The sink now enforces the same [A-Za-z0-9_-]+ fullmatch that BackupService and the queue handlers already applied — placed at the sink, so it covers every reader of that metadata field at once rather than one caller at a time.

The health endpoint was an unthrottled bcrypt oracle. location = /api/v1/health was the one auth_request location in the nginx templates without a limit_req, so an unauthenticated host on your LAN could drive one bcrypt hash (cost 12) per stored API key, per request. It now carries the same rate limit as its siblings.

Per-IP auth rate limiting now enforces the policy it declares. This one was two broken halves of a single control. proxy-public.conf blanked X-Auth-Edge-Token on exactly the public auth routes, which collapsed the app layer's per-IP login and setup buckets into one global bucket keyed to the nginx loopback peer — so any host on your network could starve your own login. Separately, the nginx auth zone rendered login_max_requests as requests per second, 60× the configured per-window policy, while every *_window_seconds setting was silently ignored and setup_max_requests never rendered at all, because both locations shared one zone. Zones now render as floored requests-per-minute, /setup gets its own zone, and burst scales with the per-window count.

Worth stating plainly, because the changelog does: the rate-limiting defect was an availability problem, not an auth bypass — X-Auth-User was never trusted from outside the edge.

Riding along: MCP read mode no longer leaks five write tools that were missing from TOOL_DEFINITIONS and fell through to bridge.execute; extract_archive() enforces the member-count, declared-total, and streamed-byte caps at the function boundary where untrusted archives actually land; credentials-file updates take a cross-process lock, so with uvicorn_workers > 1 a logout can no longer be clobbered back into validity; and a settings PATCH of allowed_origins: ["*"] with allow_credentials: true is rejected with a 422 instead of persisting and then making every subsequent boot SystemExit with no API path back in.

In plain English: a few things that were reachable from your local network, and one setting that could brick a restart, are closed.

Work that reported success while disappearing

The data-correctness fixes in this release share a property: each one logged or displayed success while losing, hiding, or overwriting something.

A task failing before handler dispatch was silently lost. The queue worker's outer try had no except, the done-callback never retrieved the task exception, and the finally removed the task from the running set while its hash still read queued. The task then existed in neither pending nor running — invisible to the reconciler and to rehydration. It never ran and never reported. Such a task is now marked failed-terminal (visible, dead-lettered) and the poller logs the exception through the canonical path.

A search-index sweep could clobber a permanently failed source back to indexed. The exhaustion branch deletes its queue row before marking the source failed, so a sibling draining later in the same batch saw zero survivors and flipped the source to indexed — hiding a source that would never be searchable. Both indexed-flip sites now refuse to overwrite a terminal failed status.

Confirming a source with a forced domain could extract under the wrong one. forced_domain and other non-None overrides were written in a second transaction after the atomic claim, while gate_decision short-circuits on the claim timestamp alone. An import analysis snapshotting inside that window saw a confirmed-but-domainless source and extracted with the auto-detected domain instead of the one you chose. Everything now rides the single write-once claim.

Alongside those: plain chat send was the only turn-enqueue path still missing the double-enqueue guard; the stuck-chat sweeper could stamp an error over an answer that had just completed; a pause landing mid-health-tick was re-attributed to health_monitor and later auto-lifted; vision page retry counters could drift permanently into completed + failed < total_pages; reset_all() could leave the app pointed at a schema that no longer existed; resetting the knowledge base could leave you with zero templates; and the migration lock file is no longer unlinked while a waiter holds it, which had let a third process lock a fresh inode and run backup-and-upgrade concurrently with an in-flight upgrade.

In plain English: several paths used to say "done" while quietly dropping or overwriting your work. They don't any more.

The API got a lot lighter per poll

None of this is a benchmark — it is a description of what the code was doing, from the changelog:

Every search ran a full 57-column source listing at page size 100,000. _get_enabled_source_ids() now uses a single-column projection through a new storage-protocol accessor, which also removes the >999-source SQLite parameter-limit hazard the old path could hit.
GET /queue/tasks shipped whole task payloads — including LLM messages arrays — on two five-second polls. The list endpoint now returns a whitelisted subset (inputs.filename, inputs.analysis_depth, operations_count). If you consume the list endpoint's data blob from your own tooling, this is the one change in this release you may need to adjust for. The detail endpoint is unchanged and still returns the full payload.
get_source read and twice-copied the entire raw upload on a three-second poll. full_text now joins the heavy-column set the response model was discarding anyway, with a narrow accessor kept for the CCX export path that genuinely needs it.
Chunk hydration batches instead of looping. Four chat and workflow tool handlers replaced per-hit get_chunk_by_id loops of up to ~100 queries per turn with a single batch fetch each, and the batch projection excludes the ~5 KB-per-hit embedding and the raw content.
Search hydration stops fetching embeddings it never reads — and a follow-up applied the same projection to twelve more batch-fetch paths, none of which read the 1024-float embedding they were hydrating.

In plain English: the screens that poll — queue, source detail, search — now ask the database for the columns they actually display.

If you run the multi-container stack

The multi-container login screen and setup wizard were broken: multi-interface-nginx.conf had drifted from its template and was missing the auth-exempt settings/public and settings/host locations, so the SPA got 401s before login. The static config is re-synced.

Also in that stack: the Valkey wipe sentinel is now written where the worker actually reads it (/data/.valkey_was_wiped, not /run/chaoscypher/...), so forced queue rehydration can fire after an AOF wipe; the boot splash ships its eight security headers, which nginx had been dropping through inherited add_header; and the production compose file no longer points LEXICON_URL at a dev-only host.

In plain English: if you run the multi-container stack rather than the all-in-one image, this release repairs login, queue recovery after a Valkey wipe, and a few config defaults.

Upgrading

No migrations apply — this release has no schema changes at all.

Drain the queue before swapping the image: stop new submissions and wait for /api/v1/queue/stats to report 0 pending on all queues. Payload-version negotiation still isn't implemented, so don't run mixed versions against the same queue.

docker pull ghcr.io/chaoscypherinc/chaoscypher:0.4.2

Or, if you run the Python packages directly:

pip install -U chaoscypher-core chaoscypher-cortex chaoscypher-neuron chaoscypher-cli

Starting fresh:

docker run -d --name chaoscypher \
  -p 80:80 \
  -p 443:443 \
  -v chaoscypher-data:/data \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/chaoscypherinc/chaoscypher:latest

Next steps

Upgrade, then reopen a chat and watch an answer stream in — that is the fix you will notice first.
If you read GET /api/v1/queue/tasks from your own tooling, check whether you depended on fields outside inputs.filename, inputs.analysis_depth, and operations_count; the detail endpoint still carries the full payload.

Full details in the changelog. ChaosCypher is AGPL-3.0 and local-first — the graph, the chat, the import and export paths all run on your own machine. Repo: https://github.com/chaoscypherinc/chaoscypher

Build a Private AI Knowledge Graph That Never Leaves Your Machine

Denis MacPherson — Fri, 04 Sep 2026 14:53:23 +0000

Every week, another AI tool asks you to upload your most sensitive documents to someone else's servers. Your contracts, medical records, internal research, personal journals -- all piped through APIs you don't control, stored in logs you can't audit, governed by terms of service that change without notice.

For a lot of use cases, that's fine. But there's a whole class of knowledge that simply cannot leave your network. Healthcare organizations bound by HIPAA. Law firms handling privileged communications. Financial institutions with regulatory obligations around client data. Companies whose competitive advantage lives in proprietary research. Or maybe you just have a journal and you'd rather not feed your inner monologue to a data center in Virginia.

The usual answer is "just don't use AI tools." That's not really an answer anymore. Chaos Cypher paired with Ollama runs a complete AI knowledge graph pipeline -- document ingestion, entity extraction, relationship mapping, semantic search, and conversational chat -- entirely on your local machine. No API keys. No usage limits. No data leaving your network. And it isn't a compromise or a toy demo: it's the same extraction pipeline, the same graph visualization, the same chat interface that works with cloud providers. You're just swapping the LLM backend from a remote API to a local one.

From Zero to Local Knowledge Graph

Here's the full workflow, start to finish. Fifteen minutes if you're following along, five if you've done this before.

Step 1: Install Ollama and pull a model.

Head to ollama.com and install it for your platform. Then pull a model:

ollama pull qwen3:30b-instruct

That downloads the model weights once. After that, Ollama runs as a local API server -- same REST interface as OpenAI, but pointing at localhost:11434.

Step 2: Start the Chaos Cypher stack.

docker run -d --name chaoscypher \
  -p 80:80 \
  -p 443:443 \
  -v chaoscypher-data:/data \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/chaoscypherinc/chaoscypher:latest

That pulls the published all-in-one image and starts it: the Cortex API server, a Neuron background worker, the web Interface, and Valkey for job queuing, all in one container. The image points the container at Ollama on your host machine through Docker's host.docker.internal bridge; the --add-host flag makes that name resolve on Linux Docker Engine (Docker Desktop resolves it on its own). No external network calls during operation -- the only downloads are the one-time Ollama model pull and a one-time fetch of the embedding model from HuggingFace at first indexing (cached afterwards; air-gapped installs can pre-seed the cache).

Step 3: Upload a document.

Open http://localhost, create a database (or use the default), and drag a PDF, DOCX, or text file into the Sources page. Chaos Cypher immediately begins indexing -- chunking the document, generating embeddings, and building a search index. This takes about 30 seconds for a 100-page PDF and requires no GPU at all (more on that below).

Step 4: Extract entities and relationships.

Indexing automatically queues entity extraction. Before the run starts, Chaos Cypher proposes the detected document domain and waits for one click of confirmation (the Review dialog, or pre-confirm in the upload wizard -- see the quickstart) -- confirm and the LLM gets to work: reading through each chunk, identifying entities (people, organizations, concepts, events), discovering relationships between them, and building a structured knowledge graph. The confirmed domain applies domain-specific extraction rules for higher quality results. For a 100-page document with a 30B model, expect roughly 5-10 minutes.

Step 5: Chat with your knowledge graph.

Once extraction finishes and the results are committed to your graph, open the Chat page and start asking questions. The chat system uses RAG (retrieval-augmented generation) to search your indexed documents and graph, then feeds the relevant context to your local LLM for a grounded answer. Everything stays on your machine -- the search, the retrieval, the generation.

Pick Your Preset

Not everyone has the same GPU. Chaos Cypher ships with VRAM presets that auto-configure the right model, context window, and batch size for your hardware. Select a preset in Settings and it handles the rest.

VRAM	Chat Model	Extraction Model	Context	GPU Examples
16 GB	Phi4 14B	Phi4 14B	16K	RTX 4080, RTX 5080
20 GB	Phi4 14B	Phi4 14B	24K	RTX A4000, RTX A4500
24 GB	Qwen3 30B	Qwen3 30B Instruct	16K	RTX 4090, RTX 3090
32 GB	Qwen3 30B	Qwen3 30B Instruct	32K	RTX 5090
48 GB	Qwen3 30B	Qwen3 30B Instruct	48K	A6000, 2x 4090
96 GB	gpt-oss 120B	gpt-oss 120B	48K	RTX 6000 Pro
128 GB	gpt-oss 120B	gpt-oss 120B	64K	DGX Spark, AMD Ryzen AI Max+ 395

The 24-48 GB presets use two Qwen3 tags -- pull both qwen3:30b (chat) and qwen3:30b-instruct (extraction) with ollama pull before your first extraction.

The sweet spot for most people is 24 GB. An RTX 4090 running Qwen3 30B gives you strong chat quality and solid extraction results. If you're on 16 GB, you'll still get a good experience for chat and search -- extraction quality will be noticeably lower on complex documents, but perfectly usable for straightforward material.

Under the Hood

A few things are worth knowing about how the local pipeline actually works.

Embeddings Are Always Local

Here's something that surprises people: the embedding model that powers semantic search runs on CPU. It has nothing to do with Ollama or your GPU. Chaos Cypher defaults to Qwen3-Embedding-0.6B, a compact model that downloads once and runs locally via sentence-transformers. Any HuggingFace sentence-transformers model can be used, and cloud providers (OpenAI, Ollama, Gemini) are also supported.

This means semantic search works even if Ollama is offline. It means you can index thousands of documents on a machine with no GPU at all. The embeddings are generated in the Neuron worker during indexing and stored in your local SQLite database (via sqlite-vec). Search queries generate an embedding on the fly, compare it against the index, and return results -- all on CPU, all local, typically in under a second.

Re-ranking also runs locally -- a compact cross-encoder reorders search results by relevance before they reach the LLM, no API calls involved. Details and model options are in the search docs.

Multi-Instance Load Balancing

Have multiple machines with GPUs, or multiple GPUs in one workstation? Configure several Ollama instances and Chaos Cypher load-balances across them (round-robin, least-loaded, or random), with independent health checks and automatic failover -- hot-reloadable from the Settings page, no restart needed. This matters most for extraction: a 500-page document produces hundreds of chunk groups, and spreading them across two or three GPUs cuts extraction time proportionally.

Thinking Mode

Qwen3-style models can reason step by step in <think> tags before answering, and Chaos Cypher detects and handles this automatically -- reasoning is separated from the final response, with graceful fallback for models that don't support it. Thinking is on by default for chat and off for extraction (where it mostly adds latency), but every VRAM preset turns it off for chat too (thinking_for_chat: false) to keep latency and VRAM headroom predictable -- re-enable it under Settings > LLM after applying a preset if you want step-by-step reasoning in chat.

Performance Reality Check

Let's be honest about the tradeoffs, because nobody benefits from hype.

Chat is great locally. Interactive question-answering with RAG retrieval works well on 24 GB+ hardware. The model has context from your documents, it generates coherent answers, latency is acceptable for interactive use. Streaming means you see tokens as they arrive -- the experience feels responsive even when total generation takes a few seconds.

Simple extraction works well. Documents with clear entity boundaries -- people's names, organization names, dates, locations -- extract reliably on local models. Legal contracts with named parties and defined obligations, research papers with cited authors and institutions, meeting notes with action items and owners.

Complex extraction is where you notice the gap. Dense academic papers with nuanced conceptual relationships, documents where entities are implied rather than stated, multi-hop reasoning about how concepts relate to each other -- this is where cloud models with 100B+ parameters still have a meaningful advantage. A Qwen3 30B model closes much of the gap, but the frontier cloud models keep a real lead on the hardest tasks. For many use cases, the local result is more than enough. For others, you'll want to use a cloud provider for the extraction pass and keep everything else local.

The good news: Chaos Cypher lets you mix and match. Use Ollama for chat and search (where privacy matters most, since those are interactive queries about your data), and use a cloud provider for the one-time extraction pass if you need maximum quality. Or keep everything local and accept the quality tradeoff. Your call.

Four Providers, One Interface

Chaos Cypher supports Ollama, OpenAI, Anthropic, and Gemini through a unified interface -- switching is a single config change, and you can mix providers per operation (Ollama for chat, a cloud model for extraction). Same extraction pipeline, same chat system, same search infrastructure either way.

Try It Yourself

Minimal configuration in settings.yaml -- easiest to set via the Settings page in the UI; for the all-in-one container the file lives at /data/settings.yaml inside the container, and for local/CLI runs it's in your platform data directory:

llm:
  chat_provider: "ollama"
  ollama_chat_model: "qwen3:30b-instruct"
  ollama_num_ctx: 32768

The default Ollama URL is http://localhost:11434; the published image
overrides it to http://host.docker.internal:11434 (via the
CHAOSCYPHER_OLLAMA_URL environment variable), which Just Works™ for the
all-in-one container talking to a host-side Ollama. On Linux Docker Engine
(not Docker Desktop), that name only resolves because of the
--add-host=host.docker.internal:host-gateway flag in the docker run
command above -- keep it, or set CHAOSCYPHER_OLLAMA_URL to your host's LAN
IP instead. To add multi-GPU instances, use ollama_instances.

Or skip the YAML entirely -- open the Settings page in the UI, select Ollama as your provider, pick a VRAM preset that matches your GPU, and you're done. The preset fills in the model name, context window, batch size, and extraction model automatically.

Then start everything:

docker run -d --name chaoscypher \
  -p 80:80 \
  -p 443:443 \
  -v chaoscypher-data:/data \
  --add-host=host.docker.internal:host-gateway \
  ghcr.io/chaoscypherinc/chaoscypher:latest

Upload a document, wait for indexing (30 seconds) and extraction (a few minutes), and you have a working knowledge graph built entirely on your hardware.

A few tips for getting the best results:

Pull models before starting Chaos Cypher. Run ollama pull qwen3:30b-instruct (or whichever models your preset uses) before your first extraction. The Neuron worker will wait for Ollama, but pre-pulling avoids the initial download delay.
Monitor VRAM usage. Run nvidia-smi to see how much VRAM your model is using. If you're near the limit, drop to a smaller context window or a smaller model. OOM kills during extraction are recoverable (the job retries), but they're slow.
Start with shorter documents. Your first upload should be a 10-20 page document so you can see the full pipeline complete in a couple of minutes. Scale up once you're comfortable with the output quality.
Experiment with extraction models. The presets pair specific extraction models with chat models. In the 24-48 GB tiers, the preset pairs the chat model with an instruct-tuned extraction variant (qwen3:30b-instruct) optimized for structured output; the other tiers use one model for both. If extraction quality isn't where you want it, try the next VRAM tier up -- the jump from 8B to 30B parameters makes a significant difference in extraction accuracy.

What's Next

Running everything locally is the starting point, not the ceiling.

If you outgrow a single GPU, the multi-instance setup lets you spread load across multiple machines on your network -- a small GPU cluster for your team, still fully private, still no cloud dependency. Configure two or three Ollama instances on different machines, point Chaos Cypher at all of them, and extraction workloads parallelize automatically.

When you do need cloud-tier quality for specific tasks, the cloud providers are there. Chaos Cypher doesn't lock you into local-only or cloud-only. You choose per-operation, per-database, whenever you want. The architecture is the same either way -- the only thing that changes is where the LLM inference happens.

The privacy argument isn't really about paranoia. It's about control. Your knowledge graph is a map of everything you know -- your research, your relationships, your institutional memory. Keeping that map on your own hardware isn't a limitation. It's a feature.

DEV Community: Denis MacPherson

ChaosCypher v0.4.3: When a Clean Restart Lost More Work Than a Crash

A clean restart was worse than kill -9

Graph edges that explained themselves by thinking out loud

One MCP sandbox bypass, and a read-only mode that wasn't

Settings that silently did nothing

The API stops freezing on Cancel All

Upgrading

Next steps

Automate Your Knowledge Pipeline: Triggers, Workflows, and AI Tools

A Concrete Workflow: Auto-Processing Research Papers

Under the Hood: 10 Built-In Tools

How Triggers Work

The "Expose as AI Tool" Feature

Workflow Portability

Try It Yourself

What's Next

From Obsidian Vault to AI-Powered Knowledge Graph in Minutes

The Workflow

Scripting the Import

A Few Tips for a Good First Pass

What's Next

Extract Smarter: How Domain-Aware AI Builds Better Knowledge Graphs

The Problem with Generic Extraction

How It Works: Upload to Knowledge Graph

Under the Hood: Domain Detection and Extraction Quality

How Detection Works

How Domains Shape Extraction Quality

Try It Yourself

What's Next

Give Any AI Assistant Direct Access to Your Knowledge Graph with MCP

What Is MCP, and Why Should You Care?

What This Actually Looks Like

Scenario 1: Research -- Connecting the Dots

Scenario 2: Coding -- Your Project's Knowledge Base in Your Editor

Scenario 3: Writing -- Summarize With Citations

Under the Hood: 31 Tools, 7 Categories

Try It Yourself

Your Data Stays Local

What's Next

Why Your RAG Chat is Missing Half the Answers (And How GraphRAG Fixes It)

What Happens When You Ask a Multi-Hop Question

Under the Hood (Technical Deep-Dive)

Personalized PageRank

Reciprocal Rank Fusion

Graceful Degradation

Try It Yourself

What's Next

ChaosCypher v0.4.2: The Streaming Bug That Only Showed Up When the Chat Was Working

The bug that hid behind a working feature

Three security fixes worth reading if you run this on a LAN

Work that reported success while disappearing

The API got a lot lighter per poll

If you run the multi-container stack

Upgrading

Next steps

Build a Private AI Knowledge Graph That Never Leaves Your Machine

From Zero to Local Knowledge Graph

Pick Your Preset

Under the Hood

Embeddings Are Always Local

Multi-Instance Load Balancing

Thinking Mode

Performance Reality Check

Four Providers, One Interface

Try It Yourself

What's Next

A clean restart was worse than `kill -9`