Codebase Memory for AI Agents: A LangGraph Pipeline That Actually Stays Accurate

I have rebuilt the same piece of infrastructure three times now: a system that lets an AI agent understand a codebase without re-reading the whole thing on every task. The first two attempts were embedding-based RAG over docstrings and they rotted within a month because nobody updated the docs when the code changed. The third attempt is the one I am describing here, and it is the first one that has survived contact with a team shipping thirty or forty commits a day.

This is not a review of somebody else’s format. It is the pipeline itself: a LangGraph state graph that watches a repo, drafts a structured knowledge file per service, scores its own output for accuracy before it ever reaches an agent, and traces every run so I can see exactly what it cost me. I am publishing the whole thing, including the parts that failed the first time.

What You’ll Find Here

• Why token-budget problems in multi-agent coding workflows are really a staleness problem in disguise
• The full LangGraph implementation: state schema, nodes, conditional edges, DynamoDB checkpointing
• A citation pass built on Amazon Bedrock (Claude Haiku 4.5) with real model IDs and cost numbers
• The piece almost nobody builds: an evaluation gate using Ragas and DeepEval that blocks a bad knowledge file from ever being published
• Full observability with self-hosted Langfuse
• A completely local version of the same pipeline using Ollama, for teams that cannot or will not send code to a cloud model
• The git hook that ties it all together
• A Production Reality Check with actual failure modes, not hypothetical ones

The Problem, Stated Precisely

Every time a coding agent starts a task on an unfamiliar repo, it has two bad options. It can read the whole repo, which burns tokens on files that have nothing to do with the task. Or it can rely on embeddings-based retrieval, which returns semantically similar chunks but no explicit answer to “what calls what” or “who owns this.”

What actually works is a third option: a small number of curated, structured files that describe each service, its responsibilities, and its dependencies, written once and kept current automatically. An orchestrator agent reads an index, decides which two or three files are relevant to the task, and only then hands them to a sub-agent. That is a token-budget decision as much as a knowledge-representation decision.

The hard part was never the file format. Markdown with YAML frontmatter is not a novel idea, and I am not going to pretend it is. The hard part is the pipeline that keeps those files honest while the code underneath them keeps moving. That pipeline is what I am walking through below.

Architecture at a Glance

                       git push / commit
                               |
                               v
                  +-------------------------+
                  | scan_diff (node) |
                  | git diff --name-only |
                  | scope to changed pkgs |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  | draft_concept (node) |
                  | Bedrock Claude Haiku |
                  | one file per service |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  | add_citations (node) |
                  | link to runbooks/PRs |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  | eval_gate (node) |
                  | Ragas faithfulness |
                  | DeepEval assert_test |
                  +-------------------------+
                          pass | | fail
                               | +----> re-draft (max 2 retries) --> human review queue
                               v
                  +-------------------------+
                  | relink (node) |
                  | fix cross-references |
                  +-------------------------+
                               |
                               v
                  +-------------------------+
                  | publish (node) |
                  | commit bundle to repo |
                  +-------------------------+

        every node emits a Langfuse trace
        every super-step checkpoints to DynamoDB

The graph is intentionally small. Five real nodes and one retry edge. The temptation with LangGraph is always to build something more elaborate than the problem needs, and I paid for that temptation on my first pass at this system, when I had eleven nodes and could not explain to a teammate why a given file existed.

The Stack, and Why Each Piece Earns Its Place

+------------------+---------------------------+--------------------------------------+
| Layer | Tool | Why this one |
+------------------+---------------------------+--------------------------------------+
| Orchestration | LangGraph | Explicit state, conditional retries, |
| | | resumable from checkpoint |
| Draft model | AWS Bedrock, Claude | Cheap enough to run on every commit, |
| | Haiku 4.5 | fast enough not to block CI |
| Checkpoint store | DynamoDB | Single-digit ms reads, TTL for expiry, |
| | (langgraph-checkpoint-aws)| survives a crashed worker mid-run |
| Eval, faithfulness| Ragas | Reference-free scoring of a generated |
| | | file against the actual code diff |
| Eval, CI gate | DeepEval | pytest-native, hard-fails a bad draft |
| | | instead of silently publishing it |
| Observability | Langfuse (self-hosted) | Per-node token/cost/latency traces, |
| | | MIT licensed, own your own data |
| Local alternative| Ollama (llama3.1:8b, | Same pipeline, zero cloud calls, for |
| | nomic-embed-text) | air-gapped or cost-sensitive teams |
+------------------+---------------------------+--------------------------------------+

Step 1: The State Schema

Everything in LangGraph starts with the shape of the state that flows between nodes. Get this wrong and every node downstream ends up guessing at what the previous node actually produced.

from typing import TypedDict, Literal
from langgraph.graph import StateGraph, END
class ConceptDraft(TypedDict):
    path: str # e.g. "services/billing-service.md"
    concept_type: str # "Service", "Runbook", "API Endpoint", ...
    title: str
    body: str # the drafted markdown body
    citations: list[str]
    faithfulness_score: float
    retry_count: int
class PipelineState(TypedDict):
    repo_root: str
    commit_sha: str
    changed_files: list[str]
    drafts: list[ConceptDraft]
    failed_drafts: list[ConceptDraft]
    status: Literal["scanning", "drafting", "evaluating", "publishing", "done", "needs_review"]

Nothing here is exotic. The field that matters most is retry_count, because it is what stops a bad draft from looping forever against the eval gate.

Step 2: Scoping the Scan to the Diff

Scanning the whole repo on every commit is what makes these pipelines too expensive to run continuously. Scoping to the diff is what makes them cheap enough to run on every commit instead of nightly.

import subprocess
def scan_diff(state: PipelineState) -> PipelineState:
    result = subprocess.run(
        ["git", "diff", "--name-only", f"{state['commit_sha']}~1", state["commit_sha"]],
        cwd=state["repo_root"], capture_output=True, text=True, check=True
    )
    changed = [f for f in result.stdout.splitlines() if f.endswith((".py", ".go", ".ts"))]
    # map file paths to the service/package that owns them
    packages = sorted({f.split("/")[1] for f in changed if "/" in f})
    state["changed_files"] = packages
    state["status"] = "drafting"
    return state

In practice I found that mapping files to packages is the one piece that is genuinely repo-specific. A monorepo with a clean services// layout needs three lines. A sprawling legacy repo needs an ownership map maintained separately, usually a CODEOWNERS file I parse instead of guessing from paths.

Step 3: Drafting with Bedrock

This is the node that actually costs money, so it is the one worth being deliberate about. Claude Haiku 4.5 on Bedrock is fast and cheap enough to run per-commit; I reserve a larger model for the human-review escalation path only, not for the routine draft pass.

import boto3
import json
bedrock = boto3.client("bedrock-runtime", region_name="us-east-1")
MODEL_ID = "us.anthropic.claude-haiku-4-5-20251001-v1:0" # cross-region inference profile
DRAFT_PROMPT = """You are documenting a software service for an AI coding agent's memory.
Given the file contents below, write a concept file with:
- A one-paragraph Responsibilities section
- A Dependencies section listing what this service calls and what calls it
- Nothing else. Do not invent dependencies you cannot see in the code.
Files:
{file_contents}
"""
def draft_concept(state: PipelineState) -> PipelineState:
    drafts = []
    for package in state["changed_files"]:
        contents = read_package_source(state["repo_root"], package)
        response = bedrock.invoke_model(
            modelId=MODEL_ID,
            body=json.dumps({
                "anthropic_version": "bedrock-2023-05-31",
                "max_tokens": 800,
                "messages": [{"role": "user", "content": DRAFT_PROMPT.format(file_contents=contents)}]
            })
        )
        body = json.loads(response["body"].read())
        text = body["content"][0]["text"]
        drafts.append(ConceptDraft(
            path=f"services/{package}.md",
            concept_type="Service",
            title=package,
            body=text,
            citations=[],
            faithfulness_score=0.0,
            retry_count=0
        ))
    state["drafts"] = drafts
    state["status"] = "evaluating"
    return state

The instruction “do not invent dependencies you cannot see in the code” is doing real work in that prompt. Haiku will confidently describe a dependency on a service that does not exist if you do not tell it not to. That single sentence cut my hallucinated-dependency rate from roughly one in six files to close to zero in testing, though I would not treat that number as a universal constant, it is specific to how much source I fed the prompt.

Step 4: The Part Everybody Skips, the Eval Gate

Here is the honest admission: a markdown file with a frontmatter type field does not know if it is telling the truth. Nothing about the format stops an LLM from writing a confident, well-formatted, wrong description of a service. If you skip this step, you are shipping a system that can silently poison an agent’s understanding of its own codebase, and it will look exactly as trustworthy as a correct one.

I use Ragas for scoring and DeepEval for the pass/fail gate, because they solve two different problems. Ragas gives me a faithfulness score, how well the draft is grounded in the actual source, without needing a hand-written reference answer. DeepEval turns that score into something CI can act on.

from ragas.metrics import faithfulness
from ragas import evaluate
from datasets import Dataset
def score_faithfulness(draft: ConceptDraft, source_contents: str) -> float:
    eval_dataset = Dataset.from_dict({
        "question": ["Describe this service's responsibilities and dependencies."],
        "answer": [draft["body"]],
        "contexts": [[source_contents]],
    })
    result = evaluate(eval_dataset, metrics=[faithfulness])
    return result["faithfulness"][0]

from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import FaithfulnessMetric
def eval_gate(state: PipelineState) -> PipelineState:
    passed, failed = [], []
    for draft in state["drafts"]:
        source = read_package_source(state["repo_root"], draft["title"])
        score = score_faithfulness(draft, source)
        draft["faithfulness_score"] = score
        test_case = LLMTestCase(
            input="Describe this service.",
            actual_output=draft["body"],
            retrieval_context=[source]
        )
        metric = FaithfulnessMetric(threshold=0.75)
        try:
            assert_test(test_case, [metric])
            passed.append(draft)
        except AssertionError:
            draft["retry_count"] += 1
            failed.append(draft)
    state["drafts"] = passed
    state["failed_drafts"] = failed
    state["status"] = "publishing" if not failed else "needs_review"
    return state

The threshold of 0.75 is a starting point, not a law. I tuned it up from 0.6 after watching two drafts pass that had subtly wrong dependency claims, and I would expect any team adopting this to spend a week watching their own false-negative and false-positive rate before trusting the number.

Step 5: Wiring the Graph, Checkpointed to DynamoDB

from langgraph_checkpoint_aws import DynamoDBSaver
checkpointer = DynamoDBSaver(
    table_name="codebase-memory-checkpoints",
    ttl_seconds=60 * 60 * 24 * 7 # expire checkpoints after a week
)
graph = StateGraph(PipelineState)
graph.add_node("scan_diff", scan_diff)
graph.add_node("draft_concept", draft_concept)
graph.add_node("eval_gate", eval_gate)
graph.add_node("relink", relink)
graph.add_node("publish", publish)
graph.set_entry_point("scan_diff")
graph.add_edge("scan_diff", "draft_concept")
graph.add_edge("draft_concept", "eval_gate")
graph.add_conditional_edges(
    "eval_gate",
    lambda s: "relink" if s["status"] == "publishing" else END,
    {"relink": "relink", END: END}
)
graph.add_edge("relink", "publish")
graph.add_edge("publish", END)
pipeline = graph.compile(checkpointer=checkpointer)

The reason DynamoDB matters here and not just “some database” is that this pipeline runs as a CI job, and CI workers get killed mid-run more often than anyone wants to admit. A checkpointed graph resumes from the last completed super-step instead of re-drafting five files because the sixth one timed out.

Step 6: Watching What It Costs You

I did not trust this pipeline until I could see exactly what each node cost in tokens and seconds. Langfuse’s @observe() decorator made that a five-minute addition, not a redesign.

from langfuse.decorators import observe
@observe()
def draft_concept(state: PipelineState) -> PipelineState:
    # same body as above, now traced automatically
    ...

Self-hosting it is a docker-compose away, which matters if your code (even just the diffs going into prompts) cannot leave your network:

git clone https://github.com/langfuse/langfuse.git
cd langfuse
docker compose up -d

Since the ClickHouse acquisition earlier this year, self-hosted Langfuse stores trace data across Postgres and ClickHouse, and the self-hosting story has if anything gotten more solid, not less. I mention this only because “will this get abandoned or paywalled” is a fair question to ask before you wire observability into a production pipeline.

What I actually watch on the dashboard: token cost per commit, faithfulness score distribution over time (a slow downward drift means my source-reading logic broke, not that the model got worse), and retry rate. Retry rate above roughly 10% has, in my experience, always traced back to a prompt problem, not a model problem.

The Local Version, No Cloud Required

Some teams I have worked with cannot send even diffs to a hosted model, full stop. The same graph runs with Ollama swapped in for the draft node, and nothing else in the architecture changes.

import ollama
def draft_concept_local(state: PipelineState) -> PipelineState:
    drafts = []
    for package in state["changed_files"]:
        contents = read_package_source(state["repo_root"], package)
        response = ollama.chat(
            model="llama3.1:8b",
            messages=[{"role": "user", "content": DRAFT_PROMPT.format(file_contents=contents)}]
        )
        drafts.append(build_draft(package, response["message"]["content"]))
    state["drafts"] = drafts
    return state

For the embedding-backed search over the published bundle (useful once you have more than a hundred concept files and index.md alone is not enough), nomic-embed-text through Ollama is the same model most local RAG setups already standardize on, so there is no new dependency to justify.

+------------------+------------------------+---------------------------+
| Component | Cloud version | Local version |
+------------------+------------------------+---------------------------+
| Draft model | Bedrock Claude Haiku | Ollama llama3.1:8b |
| Embeddings | Bedrock Titan Embed | Ollama nomic-embed-text |
| Checkpoint store | DynamoDB | SQLite (LangGraph built-in)|
| Observability | Langfuse (self-hosted) | Langfuse (self-hosted) |
+------------------+------------------------+---------------------------+

Faithfulness scores from llama3.1:8b ran roughly 10 to 15 points lower than Haiku 4.5 in my own testing on the same source files, mostly because it compresses dependency descriptions more aggressively. That gap is exactly why the eval gate exists regardless of which model drafts: you do not want to find out about a quality difference from a confused agent three weeks later.

The Git Hook That Ties It Together

#!/usr/bin/env bash
# .git/hooks/post-commit
set -euo pipefail

COMMIT_SHA=$(git rev-parse HEAD)
python -m codebase_memory.pipeline --commit "$COMMIT_SHA" --repo-root "$(git rev-parse --show-toplevel)"

I run this as a post-commit hook locally and as a required CI job on the shared branch, not just one or the other. Locally it catches drift before a PR opens. In CI it is the actual gate, because not every contributor has the hook installed, and a hook you cannot enforce is a suggestion, not a system.

Production Reality Check

Here is what I would tell someone before they build this, not after.

Cost is real but small. On a repo with roughly 40 commits a day touching an average of 2 to 3 packages per commit, the Haiku draft pass runs somewhere in the low tens of dollars a month. The eval pass roughly doubles that, since every draft gets scored against its source. That is the price of not shipping wrong documentation to an agent, and I consider it cheap for what it prevents.

Latency is the tradeoff nobody mentions upfront. A full scan-draft-eval-publish cycle for a single-package commit runs 15 to 25 seconds in my measurements. That is fine as an async CI job. It is not fine if you try to make it block a commit synchronously, and I made that mistake on the first version of this pipeline before moving it to a background job.

The eval gate will occasionally block a correct draft. Faithfulness scoring is not perfect, and a genuinely accurate but tersely written draft can score under threshold. I route anything that fails twice to a human review queue rather than silently discarding it, which is the needs_review status in the state machine above.

This is not worth building if you are not already running multiple agents against your own codebase. The entire value is in agents consuming the bundle. I built the embedding-only version first specifically because I underestimated this, and it sat unused for two months because nothing was reading it.

Package-to-file mapping is the part that will not transfer cleanly between repos. Everything else in this pipeline is close to copy-paste. That one function is the one you will rewrite for your own repo’s layout, and you should budget real time for it, not treat it as a one-liner.

Where I’d Take This Next

The eval gate right now scores faithfulness at draft time and never re-checks a published file. The obvious next step is a scheduled re-score of the whole bundle against current source, catching the case where a concept file was correct when written and became wrong three refactors later without ever triggering a re-draft. I have not built that yet. It is next.

If you have built something similar, I would genuinely like to hear where your eval thresholds landed and whether they held up past the first few weeks. That number seems to be the one everyone tunes differently and nobody publishes.

Tags: AI Agents, LangGraph, AWS Bedrock, MLOps, Software Engineering