DEV Community

bzdvdn
bzdvdn

Posted on

Why "why did our infra costs jump in Q2?" doesn't fit a graph

TL;DR: some questions don't have a fixed path through your data (search docs, hit a table, compute, verify, answer — in whatever order/combination the question needs), and drawing a graph for that class of question means either enumerating every path up front or hiding an if/else forest inside one node. ctxloom replaces the graph with typed artifacts and agents that react to their appearance — below is the same use case built both ways, side by side.

The problem

Picture a typical question from a finance lead in an internal chat assistant:

"Why did our infra costs jump in Q2?"

Answering this honestly requires:

  1. Finding relevant documents — the pricing guide, the discount policy (Confluence/docs).
  2. Pulling structured data — a CSV/table of monthly spend (GitLab/S3/DB).
  3. Computing an aggregate — not "roughly", an exact number from the table.
  4. Cross-checking textual claims against the numbers — not letting the model invent a cause the data doesn't support.
  5. Returning the answer together with proof: where each part came from.

The next question — "what if we hadn't moved to the Pro plan?" — needs a different path: a different source, a different calculation, a different verification chain. There is no universal graph for this class of questions — you can draw a graph for one specific question, but not for the class.

This is exactly what typical graph frameworks (LangGraph, CrewAI, etc.) make you pay for in complexity: either you draw a graph for every possible path up front, or you end up with a hidden branching if/else inside one node that nobody can later explain.

How this looks in ctxloom

ctxloom has no execution graph — it has artifacts (typed, versioned objects) and agents that react to their appearance. The breakdown above is just a chain of artifacts:

Question
   │
   ▼
SourceRef (ranked references to sources)
   │
   ▼
TypedDoc / Spreadsheet (lazily resolved content)
   │
   ├──► Evidence (facts extracted from text)
   │        │
   │        ▼
   │      Claim (a statement + verification against Evidence)
   │
   └──► Calculation (an aggregate computed by code, not the model)
            │
            ▼
          Answer ──supported_by──► Claim, Calculation
Enter fullscreen mode Exit fullscreen mode

Each node in this chain is a separate Produce: a small, testable unit of logic that only knows "what I need on input" and "what I create on output". Agents declare consumes/produces, and which agent runs next isn't decided by the programmer up front — the runtime figures it out from which artifact just appeared.

The repo's examples/knowledge demo has 9 agents and not a single explicit edge between them — here's the actual flow map (python -m ctxloom graph examples.knowledge.agents):

planner ──creates──► ResearchTurn ──consumed by──► search_scout, progress_evaluator
search_scout ──creates──► SourceRef ──consumed by──► resolver, table_resolver
resolver ──creates──► TypedDoc ──consumed by──► evidence_builder
evidence_builder ──creates──► Evidence ──consumed by──► verifier
table_resolver ──creates──► Spreadsheet ──consumed by──► calculator
calculator ──creates──► Calculation ──consumed by──► answer_builder
verifier ──creates──► Claim ──consumed by──► answer_builder
answer_builder ──consumes ResearchTurn──► creates Answer
Enter fullscreen mode Exit fullscreen mode

The question "how much does GPU cost in total?" goes through text search, table search, a deterministic calculation, and verification — in a single run — because each agent reacts to what appeared, not to what someone expected to see at this step.

Provenance: not "a source mentioned in the prompt", a traceable chain

The key difference from a typical RAG chatbot: the answer doesn't just reference a source in text ("according to the CSV...") — it's linked to it structurally:

answer = ctx.latest(Answer)
evidence = ctx.related(answer.id, "supported_by")[0]
Enter fullscreen mode Exit fullscreen mode

The chain Answer → supported_by → Claim → derived_from → Evidence → extracted_from → TypedDoc → materialized_from → SourceRef isn't a log line — it's a graph of relations you can walk programmatically: show "where this answer came from" in a UI, run an audit of "which documents back the last 100 answers", or find every answer that depends on a document that later turned out to be stale.

Calculations are calculated, not guessed

The model here isn't the source of truth for numbers. calculator is plain code aggregating a CSV; the LLM is used only where reasoning is actually needed (phrasing the answer, extracting a fact from text), not where arithmetic is needed. If the aggregate disagreed with what the text evidence claims, verifier catches it before the answer ever reaches the user — because checking a claim against numbers is its own deterministic Produce, not part of a prompt that "usually works out".

What this buys you in practice

  • Reproducibility — the same question against the same context version takes the same execution path; Context is versioned like git, so you can diff/rollback/replay.
  • Auditability — every derived artifact knows what it was derived from; you don't have to reconstruct the reasoning chain from logs.
  • Extensibility without rewriting the graph — a new data source is just another Source that fan_out_sources picks up automatically; there's no graph to edit because there's no graph.
  • Honest failure — if a source is unavailable or a calculation doesn't add up, a Produce returns None instead of the model "filling in" a plausible but wrong answer; the question escalates via HITL (effects.ask) if a human is needed.

Head-to-head: the same use case, two implementations

On ctxloom

No router, no shared dict, no fan-out/fan-in node. search_docs and search_table both react to the same Question independently — "GPU cost" needing both docs and the table just happens, it isn't a case anyone routes to.

(Simplified for readability into 5 agents — the real examples/knowledge demo is 9, because it also handles chat-turn routing, a lifecycle agent that decides when a turn has enough evidence to answer even if only one branch applies, and HITL escalation. Names here don't match the repo 1:1; see the link at the end for the literal code.)

from ctxloom import Consume, create_agent, produce


@produce(SourceRef)
async def search_docs(context, inputs, event, effects):
    question = find(inputs, Question)
    if question is None:
        return None
    for ref in docs_source.search(question.data.text):  # your retrieval logic
        effects.create(ref)


@produce(Spreadsheet)
async def search_table(context, inputs, event, effects):
    question = find(inputs, Question)
    if question is None:
        return None
    effects.create(load_costs_table())                  # resolves the CSV


@produce(Evidence)
async def extract_evidence(context, inputs, event, effects):
    doc = find(inputs, TypedDoc)
    if doc is None:
        return None
    for fact in extract_facts(doc.data.text):           # your extraction logic
        effects.create(Evidence(text=fact)).link("extracted_from", doc)


@produce(Calculation)
async def calculate(context, inputs, event, effects):
    sheet = find(inputs, Spreadsheet)
    if sheet is None:
        return None
    total = sum(row["cost"] for row in sheet.data.rows)  # real arithmetic, not the LLM
    effects.create(Calculation(total=total)).link("computed_from", sheet)


@produce(Claim)
async def verify_claim(context, inputs, event, effects):
    evidence, calc = find_all(inputs, Evidence), find(inputs, Calculation)
    if not evidence or calc is None:
        return None                                      # the other half hasn't landed yet — fine, we'll be re-triggered
    claim = effects.create(Claim(text=..., verified=matches(evidence, calc)))
    for e in evidence:
        claim.link("derived_from", e)
    claim.link("checked_against", calc)


@produce(Answer)
async def build_answer(context, inputs, event, effects):
    claim = find(inputs, Claim)
    if claim is None or not claim.data.verified:
        return None
    effects.create(Answer(text=claim.data.text)).link("supported_by", claim)


search_agent = create_agent("search", consumes=[Consume(Question)], produces=[search_docs, search_table])
evidence_agent = create_agent("evidence", consumes=[Consume(TypedDoc)], produces=[extract_evidence])
calc_agent = create_agent("calc", consumes=[Consume(Spreadsheet)], produces=[calculate])
verify_agent = create_agent("verify", consumes=[Consume(Evidence), Consume(Calculation)], produces=[verify_claim])
answer_agent = create_agent("answer", consumes=[Consume(Claim)], produces=[build_answer])
Enter fullscreen mode Exit fullscreen mode

verify_claim still has a two-line readiness guard — that part isn't magic, someone has to know what "ready" means. What ctxloom removes is the graph wiring around that guard: there's no join node, no conditional edge deciding who calls verify_claim, no entry in a router that has to know both search_docs and search_table exist. verify_agent just declares it consumes both artifact types; the runtime re-triggers it on either one showing up, and the guard clause is the only place "readiness" is decided — once, locally, next to the logic that uses it.

On LangGraph

To be concrete about the cost, here's roughly what the same use case takes in LangGraph. You model state as one shared, untyped dict, and you draw every path explicitly:

from langgraph.graph import StateGraph, END

class State(TypedDict):
    question: str
    doc_hits: list
    table_hits: list
    evidence: list
    claims: list
    calculation: dict | None
    answer: str | None

def route_question(state: State) -> str:
    # you write this: does the question need the table, the docs, or both?
    if looks_numeric(state["question"]):
        return "search_table"
    return "search_docs"

def search_docs(state): ...       # -> doc_hits
def search_table(state): ...      # -> table_hits
def extract_evidence(state): ...  # -> evidence, reads doc_hits
def calculate(state): ...         # -> calculation, reads table_hits
def verify_claim(state): ...      # -> claims, reads evidence + calculation
def build_answer(state): ...      # -> answer, reads claims

graph = StateGraph(State)
graph.add_node("search_docs", search_docs)
graph.add_node("search_table", search_table)
graph.add_node("extract_evidence", extract_evidence)
graph.add_node("calculate", calculate)
graph.add_node("verify_claim", verify_claim)
graph.add_node("build_answer", build_answer)

graph.set_conditional_entry_point(route_question, {
    "search_docs": "search_docs",
    "search_table": "search_table",
})
graph.add_edge("search_docs", "extract_evidence")
graph.add_edge("search_table", "calculate")
# but "how much does GPU cost in total?" needs BOTH docs and table in one turn —
# so route_question's binary choice is already wrong, and you need a fan-out/
# fan-in node instead, plus a join condition that waits for both branches.
graph.add_conditional_edges("extract_evidence", lambda s: "calculate" if needs_table(s) else "verify_claim")
graph.add_edge("calculate", "verify_claim")
graph.add_edge("verify_claim", "build_answer")
graph.add_edge("build_answer", END)
Enter fullscreen mode Exit fullscreen mode

This already shows the pain, before adding a single new question type:

  • route_question grows combinatorially. The real question — "how much does GPU cost in total?" — needs both branches, so the binary router above is already wrong on the first real input. Fixing it means a fan-out/fan-in pair: a node that dispatches to both, and a join node whose only job is re-implementing the readiness check ctxloom gets from consumes for free — except now it's graph topology, reviewed by reading edges, not by reading the function that uses the result.
  • State is an untyped shared dict, and that's a real bug source, not a style complaint. Say someone renames calculation to cost_calculation while touching calculate. verify_claim still reads state["calculation"] — no import error, no type error, no test failure until the field is None at runtime and verify_claim silently verifies against stale or missing data. In ctxloom the equivalent break is find(inputs, Calculation) returning None where a type actually changed — caught by the type checker, not by production behavior.
  • Provenance isn't a thing. state["evidence"] and state["calculation"] are just values that get overwritten; if you want "what exactly did the answer depend on", you build that bookkeeping yourself, by hand, in every node, and keep it in sync with the graph by hand too.
  • A new source or a new claim type means editing the graph. Adding a third search path (say, a ticketing system) means touching route_question, adding a node, and adding edges — not just adding a node next to the existing ones, and not without re-reading the routing logic to make sure the new branch doesn't shadow an existing one.

None of this is a knock on LangGraph for what it's built for — short, mostly-linear or a-few-known-branches flows, it's a fine fit. The mismatch is specifically the "open-ended question over heterogeneous sources, next question needs a different path" shape: a problem that doesn't have a fixed set of paths, forced into a data structure that requires one to be drawn in advance. That mismatch — and having no structural way to say "this answer is backed by exactly these facts" without hand-rolling it — is the actual reason we built ctxloom instead of building this on top of an existing graph framework: state changes already contain everything needed to decide "what runs next" and "what this depended on", so encoding that separately, by hand, as a graph plus a provenance dict, is redundant work that silently drifts from the real state the moment someone touches one side without the other.

Try it

If you're building something with a similar "no fixed path" shape, I'd like to hear about it — and if you try ctxloom, a star on the repo or an issue with what didn't work is the most useful feedback I can get.

Top comments (0)