DEV Community: Tang Weigang

Do Not Treat Pydantic AI as an Agent Magic Layer

Tang Weigang — Sat, 27 Jun 2026 06:48:39 +0000

Pydantic AI is easy to describe as another Python agent framework. That is technically true, but it misses the useful part. The project becomes interesting when you treat it as a way to make agent behavior inspectable: typed outputs, dependency injection, tool schemas, provider boundaries, traces, evals, and human approval points in one engineering surface.

The Doramagic pydantic-ai manual classifies it as an Agent SDK and runtime. The best fit is not "anyone who wants a chatbot." It is developers building observable, testable, multi-tool agent applications. The bad fit is also important: if you only need one prompt, a simple API call, or an environment where tool permissions cannot be isolated, Pydantic AI may be more machinery than you need.

My first rule for adopting it would be simple: do not start by asking the agent to do a large real task. Start by proving that a minimal agent can run with fake tools, temporary dependencies, typed output, and no unexpected side effects.

The real object is a workflow contract

The value of Pydantic AI shows up when your agent has to do more than produce text. It is most useful when:

the result needs to validate against a Pydantic BaseModel;
runtime state has to be passed through a typed RunContext;
tool arguments must be schema-checked before execution;
provider-specific behavior needs to stay behind a clear adapter boundary;
traces need to show tool choices, retries, failures, and branches;
evals need to catch regressions before release;
some tool calls require approval before they execute.

That is a different problem from "make the model answer nicely." It is closer to "make the model-driven workflow reviewable."

Structured output is not just prettier JSON

The README example pattern with output_type=SupportOutput is easy to underestimate. It is not only a formatting trick. It turns the model response into a business object that the rest of the application can inspect.

For example:

class SupportOutput(BaseModel):
    support_advice: str
    block_card: bool
    risk: int = Field(ge=0, le=10)

Now block_card is not something the caller has to infer from a paragraph. It is a field that can be reviewed, logged, tested, and gated. If the output fails validation, the framework can feed that error back to the model and retry.

The boundary is just as important: a typed field is not a policy. If risk=8 triggers a real action, the team still needs to define what risk means, who can approve the next step, and which action is allowed.

Tool calls should start with fake tools

Pydantic AI tools are normal Python functions registered through decorators, with arguments validated through schema. That is good engineering surface area. It also means the first mistake can be expensive if the tool has real permissions.

My adoption sequence would be:

register one fake tool that returns a fixed value;
pass temporary dependencies through RunContext;
verify that the agent selects the expected tool;
verify that arguments match the intended schema;
only then replace the fake tool with a real implementation.

Do not give the first agent production API keys, write access, browser automation, or a broad filesystem view. The Doramagic boundary card says the first use should start with least privilege, a temporary environment, and rollback. That is the right default.

`RunContext` is a state channel, not a junk drawer

The typed dependency pattern is one of the cleanest parts of Pydantic AI. It lets tools and dynamic instructions access runtime state without global variables. But it can also become a quiet permission leak.

I would split dependencies into three buckets:

required state: current task, current user scope, allowed data range;
read-only configuration: model choice, thresholds, feature flags;
forbidden state: long-lived secrets, full user tables, production write clients, and unrelated internal documents.

If a tool needs a user id, pass a user id. Do not pass the entire account object. If it only needs read access, do not pass a write-capable client. A stronger agent framework makes state minimization more important, not less.

Trace should explain decisions, not dump everything

Pydantic AI's observability surface is one reason it is useful for serious agent work. For agents, the trace is often more important than the final sentence. A correct answer can still hide an unsafe path: the wrong tool, too many retries, weak retrieval, swallowed errors, or a side effect that should have waited for approval.

A useful trace should answer:

Which tool was selected?
Which context or source was used?
Where did the run retry, fail, or branch?
Did structured output validation fail and recover?
Was any side effect attempted?

But observability has a data boundary. The Doramagic manual calls out community concerns around large OpenTelemetry attributes such as serialized request parameters. That is a reminder to decide what enters traces, what gets redacted, and what never gets logged.

The goal is to preserve the decision path, not to archive sensitive prompts, private documents, secrets, or oversized context.

Toolsets, MCP, and capabilities need an allowlist

Pydantic AI's toolsets, MCP support, capabilities, deferred loading, and human-in-the-loop tools are powerful because they let an agent load external capabilities in a structured way. They are also exactly where teams should slow down.

Before enabling a capability, I would write a small allowlist:

which toolsets this agent may see;
which tools are disabled by default;
which tools require approval;
what happens when a tool fails;
whether tool output can enter the final answer;
where tool calls are audited.

If an agent can see every tool all the time, you do not have a capability system. You have a permission problem waiting for the wrong prompt.

A host rule I would load first

If I were asking Claude Code, Codex, Cursor, or another AI coding host to help with Pydantic AI, I would give it this rule before asking for code:

You may help design a Pydantic AI agent, but first state:
1. whether the target is chat, RAG, a tool-using agent, or a multi-agent workflow;
2. whether output needs a Pydantic BaseModel contract;
3. which state is allowed in deps and which state is forbidden;
4. each tool's permission level, input schema, failure behavior, and approval rule;
5. which trace fields are recorded, redacted, or excluded;
6. which smoke check or eval proves the agent did not cross its boundary.

Do not claim Pydantic AI is installed or working locally without a separate run log.
Do not use real secrets, production write access, or broad filesystem access unless the user explicitly approves it.
Do not treat a prompt preview as a real project run.

That rule is more useful than "build me an agent." It turns the task into a reviewable boundary contract.

A sane first day

A practical first day would be deliberately small:

create a temporary directory;
follow the official quick start in isolation;
define one Agent with one simple BaseModel output;
add one fake tool that returns a fixed value;
pass temporary dependencies through RunContext;
capture the tool path or trace;
write a smoke check that verifies the tool call, output validation, and no unexpected side effect.

This is not flashy. It is useful because it answers the first adoption question: can this framework make my agent behavior more inspectable?

Pydantic AI's strength is not that it makes agents feel magical. Its strength is that it gives agent work a shape: types, tools, runtime state, traces, evals, approval points, and stop conditions. That is the part worth loading into an AI host.

Reference roles

Upstream project: pydantic/pydantic-ai, the source for code, installation, release behavior, and API facts, https://github.com/pydantic/pydantic-ai
Doramagic project page: an independent capability asset for AI hosts, https://doramagic.ai/en/projects/pydantic-ai/
Doramagic manual: a practical reading path for agents, providers, structured outputs, toolsets, MCP, traces, evals, and pitfalls, https://doramagic.ai/en/projects/pydantic-ai/manual/

Before You Ship an Agent, Make DeepEval Test the Failure Path

Tang Weigang — Fri, 26 Jun 2026 02:38:39 +0000

Before You Ship an Agent, Make DeepEval Test the Failure Path

Most AI agent projects add evaluation too late. The usual order is: connect the model, wire the tools, add retrieval, make the demo work, then think about evals. That is convenient, but it means the team only knows that a few happy paths looked fine. It does not know which failures are stable, which ones are dangerous, and which ones will quietly return with the next prompt or model change.

DeepEval is useful when you treat it as a release gate, not as a dashboard you add after launch. The Doramagic DeepEval manual breaks the project into the practical pieces that matter for that gate: LLMTestCase, GEval, AnswerRelevancyMetric, TaskCompletionMetric, hallucination checks, deepeval test run, deepeval generate golden, trace-aware evaluation, framework integrations, and the difference between local evaluation and Confident AI cloud synchronization.

The point is not to say "use every metric." The point is to make agent failure testable before the agent touches real workflows.

Start with failure examples

The first question should not be "which metric should we use?" A better first question is: what does a bad answer look like in this product?

For an agent or RAG system, useful failure examples might be:

the retriever found the right context, but the answer ignored the key fact;
the agent completed the task with the wrong tool;
the final answer sounded confident, but the retrieval_context did not support it;
the tool path worked once, but the trace showed repeated retries or a wrong branch;
the answer looked correct but violated a permission rule, policy rule, or user constraint.

Once those examples are written down, metrics become meaningful. Without them, a threshold such as 0.7 is just a number.

Keep the first test case boring

A minimal DeepEval test case can be very small:

from deepeval.test_case import LLMTestCase

case = LLMTestCase(
    input="What is the refund policy?",
    actual_output="Customers can get a free refund within 30 days.",
    expected_output="We offer a 30-day full refund at no extra costs.",
    retrieval_context=[
        "All customers are eligible for a 30 day full refund at no extra costs."
    ],
)

The value is not in the amount of code. The value is in separating input, actual output, expected output, and retrieval context. That separation prevents a common RAG mistake: judging whether the answer reads well instead of checking whether the allowed context supports it.

Pick metrics by failure type

DeepEval gives you several metric families. I would not start by wiring all of them into a pipeline. Pick one or two that map to a known failure.

Use answer relevancy when the answer drifts away from the user's question.
Use task completion when the agent may finish the wrong job or skip a step.
Use GEval when the product has custom criteria that need to be spelled out.
Use retrieval-aware tests when the system depends on context, sources, or documents.

The practical rule: every metric should tell you who needs to fix the failure. Is it the prompt, the retriever, the tool router, the dataset, the threshold, or the product boundary? If a failed eval only tells you "the model was bad," the eval is still too vague.

For agents, trace matters more than the final sentence

The Doramagic manual distinguishes end-to-end evaluation from trace-aware evaluation. For simple chatbots, final-answer checks already help. For agents, path quality matters more.

An agent can produce the right final sentence while taking an unsafe or unstable path. It may call a tool it should not use, retry the same step several times, continue after low-quality retrieval, or swallow a tool error and write a polished conclusion.

For agent evaluation, I would want the trace to answer four questions:

Which tool was selected?
Which context or source was used?
Where did the run retry, fail, or branch?
Which evidence supported the final answer?

Without that, you may only be evaluating writing quality.

Generated goldens still need review

deepeval generate golden is valuable because it lowers the cost of starting an eval set. It can generate candidate goldens from documents, contexts, scratch, or existing golden examples. But generated goldens are not the same thing as reviewed truth.

A safer path is:

generate 20 to 50 candidate cases;
remove duplicates, vague questions, and unsupported answers;
mark 5 to 10 cases as critical regression cases;
rerun those cases whenever the prompt, retriever, tool router, or model version changes.

That turns DeepEval into a regression habit instead of a one-time screenshot.

Local eval and cloud sync are different risk levels

The basic local path can be small:

pip install -U deepeval
deepeval test run test_chatbot.py

If you log in and sync reports, datasets, traces, or production monitoring to Confident AI, you should treat it as a separate data-boundary decision. Before doing that, answer:

will inputs, outputs, retrieval context, or traces be uploaded?
do any cases contain user data, internal documents, or secrets?
who can view the report?
can failure cases be redacted?
should CI be allowed to sync results automatically?

This is not a criticism of DeepEval. It is just the normal boundary work for any eval or observability system.

A useful host rule

If an AI coding host is going to help set up DeepEval, I would give it this rule first:

You may design DeepEval tests, but first state:
1. whether the target is RAG, an agent, a chatbot, or a single prompt;
2. the failure examples being tested;
3. which metric maps to which failure;
4. why the threshold is chosen;
5. whether trace-aware evaluation is needed;
6. whether cloud sync, API keys, or user data are involved.

Do not treat LLM-as-a-Judge scores as absolute truth.
Do not treat generated goldens as human-reviewed labels.
Do not claim DeepEval is installed or validated locally without a separate run log.

That rule is more useful than "add evals." It makes the evaluation plan reviewable.

A sane first day

For a first run, I would pick one real workflow and write ten cases:

three normal success cases;
three likely hallucination cases;
two missing-context cases;
one permission-boundary case;
one empty-result case.

Then I would add one metric and make the failure explanation useful before adding more metrics, trace collection, generated goldens, or CI gates.

DeepEval's value is not that it makes AI systems look controlled. Its value is that it makes failure earlier, sharper, and easier to reproduce.

Reference roles

Upstream project: confident-ai/deepeval, the source for code, installation, releases, and API facts, https://github.com/confident-ai/deepeval
Doramagic project page: an independent capability asset for AI hosts, https://doramagic.ai/en/projects/deepeval/
Doramagic manual: a practical reading path for test cases, metrics, tracing, generated goldens, pitfalls, and boundaries, https://doramagic.ai/en/projects/deepeval/manual/

Before an Agent Uses Qdrant, Write the Retrieval Contract

Tang Weigang — Thu, 25 Jun 2026 02:25:03 +0000

Before an Agent Uses Qdrant, Write the Retrieval Contract

When people wire a vector database into an agent, the first milestone is usually simple: store embeddings, run a search, pass the hits back to the model. That is enough for a demo, but it leaves the most important questions implicit. Which collection was searched? Was the payload filter mandatory or optional? Is the score threshold a business rule or a temporary guess? If nothing is returned, does that mean the knowledge is missing, the filter was too strict, the embedding model drifted, or the index is not ready?

The Doramagic Qdrant project pack is a useful reminder that Qdrant is not just a "semantic search endpoint." Its manual points at HNSW, sparse and multivector search, quantization, payload indexing, filtering, sharding, replication, WAL, and storage internals. That shape matters when an AI host is going to call Qdrant as a tool.

The thesis is simple: Qdrant should enter an agent workflow as a retrieval layer with a contract, not as a magic memory button. The Doramagic pack is not a prompt library. It is an independent capability asset: the Human Manual gives a reading route, the Prompt Preview gives pre-install host instructions, the pitfall notes and boundary card define limits, the source map points back to repo evidence, and eval or smoke-check thinking turns "it sounds right" into something reviewable.

This is a technical workflow note, not an official Qdrant guide and not a claim that Qdrant was installed in this environment.

Do not stop at "the vector insert worked"

For an agent, the risk is rarely that Qdrant has no search API. The real risk is that the agent treats retrieval as a black box and turns a plausible hit into a confident answer.

Before giving Qdrant to an agent, I would make the agent state five things for every retrieval call:

retrieval intent: fact lookup, similar case, code snippet, or candidate recall;
collection scope: one collection, several collections, or a routed search;
payload filters: tenant, permission group, document type, version, and time window;
score use: top-k, threshold, reranking, and who owns each decision;
empty-result handling: missing knowledge, bad filter, stale index, or embedding mismatch.

If these are not explicit, the model can accidentally turn "nearest neighbor" into "verified fact."

Treat payload filters as permission boundaries

In small RAG demos, filters often look like convenience. In a real system, payload filters are often access control.

If your payload includes fields like tenant_id, acl_group, source_type, or document_version, they should not be optional knobs that the agent can drop when it wants more recall. They should be part of the tool contract.

One plain rule is enough:

Every user-facing retrieval call must include tenant, permission, and version filters.
If those fields are missing, the tool should refuse or return a structured "filter_required" error.

This is not elegant, but it prevents a familiar failure: the agent widens the query because it is trying to be helpful.

Vector score is not answer confidence

A Qdrant score says something about similarity under a particular representation. It does not prove the source is current, authorized, complete, or correct.

I prefer to make the agent keep two layers separate:

retrieval evidence: point id, source id, payload, collection, filter, score;
answer evidence: which retrieved items were used, which were rejected, and why.

That separation makes the output slightly more verbose, but it catches a common bug: a high-scoring stale chunk becomes the basis for a very confident answer.

Quantization and multivectors need a test set

Qdrant exposes serious retrieval machinery: quantization, sparse vectors, dense vectors, multivectors, and payload indexes. Those features are not automatic upgrades. They change memory use, latency, recall, and debugging behavior.

Before asking an agent to optimize the retrieval setup, I would build a tiny acceptance set:

five questions that should retrieve the right source;
three questions that are likely to retrieve the wrong neighbor;
two permission or version-boundary questions;
one empty-result question.

Run this before changing quantization, hybrid search, reranking, or collection structure. Otherwise you may improve a benchmark while making your actual agent less reliable.

A minimal retrieval contract

The contract I would give an AI host is short:

You may use Qdrant for candidate retrieval, but every call must state:
1. retrieval intent;
2. collection and payload filter;
3. top-k, score threshold, and rerank setting;
4. which returned fields may be used in the final answer;
5. how to handle empty, low-score, or missing-permission results.

Do not treat vector similarity as factual confidence.
Do not broaden a query when tenant, permission, or version filters are missing.
Do not claim Qdrant has been installed or validated locally unless a separate run log proves it.

That contract turns retrieval into a reviewable step instead of an invisible tool call.

A safer first run

If I were starting today, I would not begin with a large import. I would create a small collection with 20 to 50 source-controlled samples and complete payload fields. Then I would expose only a read-only search tool and require the agent to output the retrieval plan, raw evidence, and answer citation.

After that loop is stable, I would add writes, bulk import, quantization, multivectors, and more aggressive routing.

Qdrant can be a strong retrieval layer. The AI host still needs a contract for scope, permissions, and failure interpretation.

Reference roles

Upstream project: Qdrant repository, the source of code, API, and release facts, https://github.com/qdrant/qdrant
Capability asset: Doramagic Qdrant project page, an independent capability resource pack for AI hosts, https://doramagic.ai/en/projects/qdrant/
Guide and validation notes: Doramagic Qdrant manual, useful before loading the pack into an agent workflow, https://doramagic.ai/en/projects/qdrant/manual/

Before an Agent Runs Code in E2B, Define the Sandbox Contract First

Tang Weigang — Wed, 24 Jun 2026 02:26:14 +0000

E2B is easy to describe too quickly: give an AI agent a secure sandbox so it can run code.

That description is useful, but it skips the part I would care about before connecting it to Claude, Codex, Cursor, or a custom agent:

What is the agent allowed to touch inside the sandbox, and what must happen when the run fails?

This note is based on the independent Doramagic E2B manual:

https://doramagic.ai/en/projects/e2b/manual/

Project page:

https://doramagic.ai/en/projects/e2b/

It is not an official E2B document. I am using it as a pre-adoption checklist. One important caveat from the local Doramagic pack: the test log says real host dogfooding and runtime install evidence have not been executed. So this is not a production-readiness claim. It is a boundary note for the first safe evaluation.

1. Treat E2B as a sandbox contract, not a permission switch

The useful promise is clear: E2B gives developers isolated cloud sandboxes where AI agents and apps can execute code, run commands, process data, and manage files.

That is exactly why the first question should not be "can it run code?"

The first question should be:

which commands are allowed;
which paths can be read and written;
whether network access is allowed;
whether dependencies may be installed;
whether credentials are allowed at all;
how stdout, stderr, and non-zero exit codes are handled;
how artifacts are exported;
how the sandbox is torn down.

Without that contract, the upstream tool may be fine while the agent workflow is still unsafe.

2. The first run should be disposable and boring

The official first install entry recorded in the Doramagic pack is:

npm i e2b

I would not start by attaching that to a real project. My first E2B check would be intentionally small:

Create one disposable sandbox.
Use no secrets.
Use a tiny input.
Allow one command.
Write only under /tmp/e2b-first-run/.
Set a fixed timeout.
Export one small artifact.
Destroy the sandbox.
Record stdout, stderr, exit code, and cleanup status.

That may sound slow, but it answers the operational question that matters: can the agent complete one reversible run without widening the boundary?

If that first fixture is unstable, adding templates, network calls, persistent volumes, or MCP servers will only make debugging harder.

3. Command execution needs a failure contract

The E2B manual describes command execution as returning stdout, stderr, exit code, and optional error information. Non-zero exits can become exceptions.

That means the host agent needs rules for failure, not just success.

A practical first contract:

non-zero exit code stops the workflow;
stderr is summarized before being fed back to the model;
generated files are listed before export;
no command may read outside the allowed working directory;
no environment variable is printed unless explicitly approved;
timeout is treated as a failed run, not as a reason to retry with a larger scope.

The most common failure in agent tooling is not a dramatic exploit. It is a quiet expansion of scope: one more command, one more path, one more network call, one more unreviewed artifact.

4. Do not enable templates, network, volumes, and MCP at the same time

E2B has more surface area than "run a command." The manual covers SDKs, templates, filesystem operations, network behavior, ready commands, persistent volumes, and MCP server integration.

Those are useful features. They should not all be part of the first test.

My order would be:

start and destroy a default sandbox;
run one harmless command;
write one small file under a controlled path;
export one artifact;
add a template only after the basic path works;
add network only when the task requires it;
add persistent volume or MCP only after cleanup and access boundaries are clear.

This keeps failures legible. If a first run already includes template build, network, file upload, ready command, and long-running process behavior, the failure will be difficult to assign.

5. Use the pitfall log as a test plan

The Doramagic pitfall log lists source-linked issues such as auto-paused processes, closed port errors, template creation failures, build polling timeouts, API key authorization problems, file/template confusion, and paused sandbox persistence questions.

I would not present those as guaranteed current bugs. Some may have changed by version.

I would turn them into checks:

after pause/resume, do process and file states match the expected contract?
when a port is unavailable, does the run fail with a bounded timeout?
when template build fails, does the agent stop instead of guessing a workaround?
when an API key is wrong, is the key kept out of logs?
after export, is the sandbox actually destroyed?

That is the value of the manual: it gives a better first-run checklist than a feature list.

6. A safe AI-host handoff

If I were handing E2B context to an AI coding host, I would not only paste the project link. I would add an instruction like this:

You may consider E2B as a candidate sandbox runtime.
Do not install anything yet.
First return a go/no-go review:
- whether this task actually needs code execution;
- whether it needs network;
- whether it needs filesystem access;
- whether credentials are involved;
- the smallest reversible verification fixture;
- timeout, cleanup, and artifact export plan;
- missing evidence that must stop the run.
Do not run install commands, read private local files, or use real API keys unless the command and rollback plan are approved.

That wording is not bureaucracy. It keeps the model from turning "sandbox exists" into "everything inside the sandbox is safe."

7. My working conclusion

E2B is interesting for agent workflows because it can give code execution a controlled runtime.

But the adoption question is not "can my agent run code now?"

The better question is:

Can my agent run one tiny task, with no secrets, a known command boundary, fixed timeout, inspectable output, and a cleanup path?

I would not move to real workloads until that answer is yes.

Do not treat LangGraph as a longer chain: define state, interrupts, and recovery first

Tang Weigang — Tue, 23 Jun 2026 06:55:06 +0000

The easiest way to misunderstand LangGraph is to see it as “LangChain, but with more steps.”

That misses the point.

LangGraph becomes useful when an agent is no longer a single prompt or a simple chain. It becomes useful when the workflow has state, branches, tool calls, human approval, checkpointing, and recovery behavior that must be inspected before the agent is trusted inside a real AI host.

I used the Doramagic LangGraph manual as the source-backed reading layer for this note:
https://doramagic.ai/en/projects/langgraph/manual/

This is an independent project guide, not an official LangGraph document. I use it as a pre-adoption checklist: what should be understood before wiring a project into Claude, ChatGPT, Cursor, Codex, or another AI host.

The point is not to create another prompt library. The useful artifact is a capability resource pack: a manual, source map, boundary notes, pitfall log, smoke check, lightweight eval criteria, feedback notes, and host-ready context that help a developer decide what to verify before adoption.

1. The real boundary is State, not the prompt

For a one-shot model call, the prompt is often the main boundary.

For LangGraph, the first boundary is the State schema:

which fields move between nodes;
which fields a node may update;
how concurrent branches merge values;
which values enter a checkpoint;
which values should never be persisted.

This is why reducers matter. A message list is usually not just overwritten. It needs an append or merge rule such as add_messages or the TypeScript equivalent. That small implementation detail decides whether parallel work preserves context or silently drops it.

My preferred first run is not a “universal agent.” It is a tiny graph with one State schema, one node, one partial update, and one explicit reducer. If that is not clear, adding tools will only hide the problem.

2. `compile()` is the boundary between description and runtime

Before compile(), a LangGraph graph is a description: nodes, edges, conditional routes, state fields, reducers.

After compile(), the Pregel-style runtime takes over. Nodes execute in supersteps. Partial state updates go through channels and reducers. Conditional edges choose the next node.

That changes how debugging should work. When a graph fails, do not inspect only the node function. Inspect four contracts at the same time:

Does the State schema allow this node to write the returned key?
Does the node return at least one valid State field?
Does the reducer match the intended merge behavior?
Does the conditional edge have a real exit path?

The Doramagic manual highlights errors such as InvalidUpdateError: Must write to at least one of [...]. That is not just a random runtime annoyance. It often means the node return shape and State schema do not agree.

3. Human-in-the-loop is an execution contract

LangGraph’s interrupt and human-in-the-loop support should not be treated as interface decoration.

If an agent can send email, edit files, call external APIs, or touch production systems, the approval step should be part of the graph contract, not a person watching the screen and hoping to catch mistakes.

A practical pattern is:

the model proposes a tool call;
a graph node raises an interrupt;
the human approves, ignores, or edits one concrete action;
the structured response returns to the graph;
execution resumes from the interrupt point instead of restarting the whole workflow.

That is why LangGraph fits recoverable workflow agents better than simple chat demos.

4. Checkpointing is not logging

Checkpointing is easy to describe as “saving history.” In LangGraph, that description is too weak.

The manual separates checkpointing, serialization, and stores:

a checkpointer is scoped to a thread and supports durable execution, replay, and resume;
a store is cross-thread long-term memory;
serialization defines how Python objects become stored and reconstructed data.

One security detail deserves attention before adoption: the checkpoint serializer can handle Python objects from checkpoint data. New applications should consider LANGGRAPH_STRICT_MSGPACK=true or an explicit allowed_msgpack_modules list for JsonPlusSerializer.

That is not an advanced edge case. If checkpoint data may cross a trust boundary, deserialization policy is part of the system boundary.

5. A better first-run checklist

For a first LangGraph test, I would keep the path intentionally small:

Use a temporary directory. Do not connect production data.
Define a minimal State with messages and one business field.
Write one node that returns only valid State fields.
Attach a reducer to any append-like field.
Add one interrupt around a concrete tool-like action.
Add a checkpointer.
Force one node to fail, then verify resume behavior.

If this path fails, do not move on to real tools yet. The problem is not that the agent is weak. The runtime boundary has not been verified.

The smoke check should have pass/fail criteria, not a vague “seems to work” result. A useful minimum is: the node writes only declared State fields; the interrupt happens before a real side effect; resume behavior does not repeat already successful sibling writes; and serializer limits are explicit. If any of those are unclear, the decision should be HOLD, not “try it in production and see.”

6. My practical take

LangGraph is worth studying because it makes the hard parts of agent workflows explicit:

state;
branching;
tool calls;
human approval;
checkpointing;
failure recovery;
thread-scoped state versus long-term memory.

If your task is a one-off model call, LangGraph may be more machinery than you need.

If your AI host needs to run multi-step work where each step must be inspectable, pausable, recoverable, and reviewable, LangGraph becomes much more interesting.

The better question is not “Can LangGraph build an agent?”

The better question is: “Can my agent’s state, tool boundary, and recovery path be written as a graph?”

Answer that first. Then decide whether LangGraph belongs in the stack.

Do not add LLM evals after launch: use promptfoo to define the failure boundary first

Tang Weigang — Mon, 22 Jun 2026 02:57:11 +0000

A common LLM workflow mistake is to tune the prompt first and add evaluation later. That order feels fast, but it leaves the team with a weak release boundary. When the workflow fails in production, nobody can say exactly which behavior should have blocked the release.

The category claim matters: Doramagic is not a prompt library and not a README summary. It turns an open-source project into a portable AI agent capability asset: source map, host instructions, prompt preview, pitfall log, eval or smoke check, boundary card, human manual, test log, and feedback path.

The useful way to read promptfoo is not as a leaderboard tool. It is a contract layer for LLM workflows: before changing prompts, switching models, adding tools, or giving an agent more authority, define what must pass, what must fail, and what evidence is enough.

The Doramagic promptfoo manual highlights four surfaces that matter in practice.

1. An eval is not a score. It is a reproducible judgment set.

Promptfoo's core engine ties together configuration, provider calls, assertion grading, and result aggregation. YAML or JSON config can place prompts, test cases, providers, and assertions in one run.

That changes the question from "did the model sound good?" to a more useful checklist:

Which inputs must pass every time?
Which outputs must be rejected?
Which tool-call structures must be valid?
Which checks should be deterministic, and which deserve an LLM-as-judge rubric?
After a prompt, provider, or model change, which cases must run again?

If those answers are not in the eval config, the final score is mostly decoration.

2. Agents can run evals, but the scope must be explicit.

One important detail in the manual is promptfoo's MCP tool surface. Tools such as list_evaluations, get_evaluation_details, run_evaluation, and share_evaluation allow AI agents to drive evaluations programmatically.

That is powerful, but it needs a boundary. run_evaluation accepts a config path, optional test-case filters, prompt and provider filters, concurrency settings, timeout limits, and pagination controls. An agent should not treat this as a casual "run everything" button.

Before running an eval, the agent should state which config it will use, which cases it will run, how much concurrency it needs, what timeout applies, and why this run is the right evidence for the current change.

3. Many providers do not mean automatic portability.

Promptfoo supports a broad provider ecosystem: OpenAI, Anthropic, Google Vertex, xAI, Bedrock, Cerebras, agent SDKs, MCP tools, and custom gateways. That breadth is useful because one test set can compare different execution surfaces.

But every provider has its own behavior around structured outputs, tool calls, timeouts, caching, permissions, and cost. A test that passes with one provider should not be treated as proof that another provider is safe.

For a first run, I would not start with a giant model comparison. I would fix one provider and one real workflow, run a small smoke eval, then add the second provider only after the assertions are stable.

4. Red teaming should become a negative-case release contract.

Promptfoo's redteam layer reuses the evaluation engine and adds adversarial providers, target invocation, judging, and iterative feedback. The manual notes presets around prompt injection, harmful content, PII, and related risks.

The mistake is treating red teaming as a one-off report. The useful version is a release gate: which prompt-injection attempts must fail, which PII outputs must be blocked, and which tool permissions must be denied before the workflow ships.

If the redteam output does not feed CI, a release checklist, or a human review step, it is only a demo.

A safer first-run path

For an AI coding agent, I would use promptfoo like this:

Read the Doramagic manual and identify the runtime boundary.
Create a minimal promptfoo config in a temporary directory.
Use one provider and one real workflow first.
Run 3 to 5 high-value test cases before expanding the suite.
Start with deterministic assertions, then add LLM rubrics only where judgment is unavoidable.
Report passing examples, failing examples, cost, latency, and what changed.
Do not claim the workflow is ready until failed cases are reviewed.

The Doramagic promptfoo pack does not replace the upstream docs. Its job is to make promptfoo loadable by an AI coding host as an operating contract: read the manual, check the pitfalls, run the evals, and only then recommend a prompt or model change.

The feedback loop is part of the asset. If the first run exposes a new failure case, it should update the pitfall log, eval suite, boundary card, or human manual. Otherwise the article is just content; the capability asset is what lets the next agent start from better evidence.

Doramagic manual: https://doramagic.ai/en/projects/promptfoo/manual/
Doramagic project page: https://doramagic.ai/en/projects/promptfoo/
GitHub pack: https://github.com/tangweigang-jpg/doramagic-promptfoo-pack
Upstream project: https://github.com/promptfoo/promptfoo

Disclosure: this is an independent Doramagic project asset, not an official upstream release.

Before You Let an Agent Convert Documents, Narrow the MarkItDown Boundary

Tang Weigang — Mon, 22 Jun 2026 00:05:11 +0000

MarkItDown looks like a simple utility at first glance: point it at a PDF, Word file, spreadsheet, image, HTML page, archive, audio file, or URL, and get Markdown back.

For an AI agent, that simplicity is exactly where I would slow down.

Once a coding agent or MCP host can call a document converter, the real question is not "can it convert files?" The real question is:

What is the smallest input surface the agent may touch, and what evidence proves the Markdown output is good enough for the next step?

This note is based on the independent Doramagic MarkItDown manual:

https://doramagic.ai/en/projects/markitdown/manual/

It is not an official Microsoft or MarkItDown document.

Use it for LLM ingestion, not layout reconstruction

MarkItDown is useful because Markdown is close to plain text while still preserving headings, lists, links, and some structural cues. That makes it practical for retrieval, summarization, indexing, and agent workflows.

But I would not treat it as a high-fidelity document renderer.

The first boundary check is plain:

If the task needs a human-perfect copy of a PDF layout, hold.
If the task needs searchable text with inspectable structure, continue.
If the task involves untrusted uploads or remote URLs, define the allowed paths and schemes before any agent runs it.

That distinction prevents a bad handoff where the agent reasons over Markdown that was never fit for the downstream claim.

Start with one format, not every optional dependency

The convenient install path is:

pip install "markitdown[all]"

That is fine for exploration. It is not always the best first production-style test.

For a first agent workflow, I prefer a narrow install:

python -m venv .venv
source .venv/bin/activate
pip install "markitdown[pdf,docx]"
markitdown sample.pdf -o sample.md

Then verify the artifact:

the Markdown file exists and is non-empty;
headings and links survived well enough for the task;
tables are manually sampled instead of trusted blindly;
the command, package version, input path, and output path are recorded;
the agent is instructed to read the Markdown artifact, not go back and inspect arbitrary source files.

That small test catches the most common mistake: calling "installation works" the same thing as "the document pipeline is safe."

PDF and OCR are where expectations drift

The Doramagic manual is most useful where it keeps the limits visible.

PDF conversion is best effort. Complex tables, headers, footers, multi-column layouts, scanned pages, and image-only documents can lose structure. The output may still be useful for search or rough summarization, but it should not become a source of record without sampling.

The markitdown-ocr plugin extends the system with LLM Vision OCR for embedded images and scanned PDFs. That can be the right tool, but it changes the operating model:

OCR may add model cost per page or image.
If no llm_client is provided, the plugin can load while OCR is skipped and the standard converter is used.
OCR output should be sampled before an agent uses it for legal, financial, medical, or compliance-sensitive reasoning.

My first-run decision rule is:

GO: controlled DOCX/PDF input, Markdown generated, sample checks pass.
HOLD: tables or scanned pages are readable but not structurally reliable.
NO-GO: sensitive scanned documents are pushed into an agent workflow before OCR quality and access boundaries are checked.

Treat the MCP server as a local tool, not a public service

MarkItDown also has an MCP server package. That is useful when an MCP-capable host needs a document-to-Markdown tool.

The safe default is local and narrow: trusted local agents, localhost binding, a small mounted work directory, and a clear rule for remote URLs.

A first instruction I would actually use:

Allow MarkItDown MCP to convert only files under /workdir/inbox/.
Do not fetch external URLs.
Write Markdown outputs under /workdir/out/.
Record input path, output path, file size, and MarkItDown version.
Stop on scanned PDFs, archives, remote URLs, empty output, or unknown extensions.

That turns MarkItDown into a bounded capability. Without that boundary, a document converter can quietly become a broad file and URL access tool.

A compact acceptance checklist

Before I let an agent rely on MarkItDown output, I want these checks visible:

Check	Acceptance rule
Install scope	Use only needed extras, or explain why `[all]` is required.
Input boundary	Restrict to known directories or approved URLs.
Output evidence	Save Markdown and inspect headings, links, lists, and table samples.
PDF caveat	Mark complex layouts and scanned pages as lower confidence.
OCR path	Record whether `markitdown-ocr`, model client, and sampling are enabled.
MCP exposure	Keep local by default; do not expose a converter server without auth and network controls.
Failure handling	Empty output or unsupported format stops the agent workflow.

The practical value of MarkItDown is not that it magically understands every document. It gives an agent a narrow path from messy files to inspectable text. The narrower that path is on day one, the easier it is to trust the next step.

Sources:

Doramagic MarkItDown manual: https://doramagic.ai/en/projects/markitdown/manual/
Doramagic project page: https://doramagic.ai/en/projects/markitdown/
Official repository: https://github.com/microsoft/markitdown

Disclosure: this is a practitioner note based on an independent Doramagic capability manual and public MarkItDown repository material. It is not affiliated with or endorsed by Microsoft unless explicitly stated.

Before You Add Memory to an AI Agent, Decide What the Agent Is Allowed to Remember

Tang Weigang — Sun, 21 Jun 2026 11:44:28 +0000

Before You Add Memory to an AI Agent, Decide What the Agent Is Allowed to Remember

Memory is one of those agent features that sounds obvious until it is connected to a real system.

The naive version is simple: save the conversation, retrieve something similar later, and call it memory.

The operational version is harder:

What is short-term conversation context?
What is long-term user or domain knowledge?
What is a reasoning trace?
Who owns a remembered entity?
Which memory can be reused across sessions?
Which memory should expire, be corrected, or stay private?
How do you prove that the agent used the right memory instead of a convenient hallucination?

That is the useful way to read agent-memory, the Neo4j Labs project covered in the Doramagic manual.

It is not just a "vector store for agents". The manual frames it as a graph-native memory layer for AI agents and context graphs, backed by Neo4j, with Python and TypeScript SDK surfaces and a hosted NAMS backend option.

The first useful mental model: three memory tiers

The most important part is the separation of memory types.

The Doramagic manual describes three main tiers:

Tier	What it holds	Why it matters
Short-term memory	Session or conversation message history	Keeps the current turn grounded without pretending every message is permanent knowledge.
Long-term memory	Entities, preferences, relationships	Lets the system remember durable facts, but also creates privacy and correction obligations.
Reasoning memory	Steps, tool calls, traces, similar traces	Makes the agent's behavior reviewable instead of turning memory into an invisible black box.

That split is practical because "remember everything" is not an implementation strategy. It is a data governance problem wearing a friendly product name.

If an AI host is going to use memory, the host should know which tier it is touching.

A user message might belong in short-term memory.

A confirmed customer preference might belong in long-term memory.

A failed tool call and recovery path might belong in reasoning memory.

Those are different records with different risks.

The graph part is not decoration

agent-memory uses Neo4j as the backing graph. That matters because agent memory is rarely just a bag of text chunks.

Useful memory often has structure:

a person belongs to an organization
a task was requested in a session
a tool call touched an entity
a preference applies to one user but not another
a reasoning trace created or updated a record

The manual highlights POLE+O entity typing: PERSON, ORGANIZATION, LOCATION, EVENT, and OBJECT, plus extension entity types. That gives the memory system a vocabulary for durable knowledge instead of treating every remembered thing as the same kind of note.

The result is not automatically safe or correct. It is just more inspectable.

That is the point.

Backend choice changes the operating boundary

The manual describes two backend paths:

direct Neo4j through Bolt
hosted NAMS through a REST backend

That is not a small deployment detail. It changes the boundary you need to check.

With a local or self-hosted Neo4j path, you are responsible for database configuration, tenant isolation, backups, and operational access.

With NAMS, you get a hosted memory service path and ontology surfaces, but now the remote service boundary, workspace ownership, and API configuration matter.

The practical first question is not "which one is better?"

The practical first question is:

Where is the memory allowed to live, and who can read it later?

If you cannot answer that, do not let an agent write durable memory yet.

Ontology is the part teams will underestimate

The manual calls out a typed, versioned ontology layer in NAMS. This is more important than it sounds.

Without an ontology boundary, agent memory can quietly drift:

the same entity appears under multiple names
preferences become mixed with facts
tool results become treated as user intent
stale knowledge remains in retrieval because nothing marks it as old
private and shared memory end up in the same retrieval pool

An ontology does not solve those problems by itself, but it gives the team a place to define what is valid.

For a first run, I would not start by building a complex domain ontology.

I would start with a deliberately small one:

one user
one session
two entity types
one relationship type
one trace
one correction case

If that cannot be inspected and corrected, scaling the memory system will only make the failure harder to see.

The first safe verification run

Before wiring agent-memory into a serious AI workflow, I would run a small sandbox test.

The test should not require production credentials or real user data.

A good first run looks like this:

Create a temporary test user and session.
Add a small conversation message to short-term memory.
Add one explicit long-term entity, such as a fake preference.
Record one reasoning step or tool call.
Retrieve context on the next turn.
Verify which memory tier produced which returned item.
Correct or delete one memory record.
Confirm the correction is visible in the next retrieval.

The key artifact is not the demo output.

The key artifact is the audit trail:

what was stored
why it was stored
where it lives
how it is retrieved
how it is corrected
what the agent is not allowed to remember

The main pitfall

The biggest mistake is treating memory as a feature toggle.

"Add memory" sounds like a product improvement.

In practice, it changes the agent's state model.

A stateless agent can be wrong in one run.

A stateful agent can be wrong, remember the wrong thing, and use that wrong memory later with confidence.

That does not mean memory is bad. It means memory needs a smaller first run, clearer permissions, and a visible review path.

A practical adoption checklist

Before giving an AI host access to a memory layer, answer these questions:

What memory tiers are enabled?
Which writes are automatic and which require approval?
Where is the backing store?
Is memory scoped by user, workspace, tenant, or project?
Can a user inspect and correct remembered facts?
Are reasoning traces stored separately from durable user knowledge?
Does retrieval show provenance?
Is there a deletion path?
Is there a sandbox test that proves all of the above?

If the answer is unclear, the next step is not production integration.

The next step is a smaller verification loop.

Reference: Doramagic agent-memory project page and manual: https://doramagic.ai/en/projects/agent-memory/manual/

Disclosure: this post is based on an independent Doramagic project pack for neo4j-labs/agent-memory. It is not official Neo4j documentation and does not imply endorsement by the upstream project.

Before You Give an AI Agent a Browser, Define the Puppeteer Boundary

Tang Weigang — Fri, 19 Jun 2026 03:51:39 +0000

Before You Give an AI Agent a Browser, Define the Puppeteer Boundary

Puppeteer is one of the most practical tools you can give an AI coding agent. It lets a Node.js workflow control Chrome or Firefox through a high-level JavaScript API, which makes it useful for browser automation, screenshots, scraping, page checks, and repeatable web tasks.

That power is also the risk.

Once an agent can open a browser, it may touch live web sessions, read page content, follow links, download files, submit forms, or capture screenshots that contain private state. The first question should not be "Can the agent use Puppeteer?" The better question is:

What is the smallest browser task the agent can run while producing evidence and staying inside a clear boundary?

This is the first-use checklist I would apply before loading a Puppeteer-oriented capability into an AI coding host.

1. Separate the Two Install Paths

Puppeteer has two common install paths:

npm i puppeteer: the full package, including browser download behavior.
npm i puppeteer-core: the lighter core library, where you provide the browser executable yourself.

That choice matters for agent workflows.

If the agent is only checking a known local page or a CI preview, puppeteer-core plus an explicit executablePath may be easier to reason about. If the agent needs a bundled browser path for a quick isolated smoke test, the full package can be convenient, but it also changes the install surface.

Do not let the agent choose this casually. Ask it to state:

which package it wants;
whether browser download is expected;
where temporary files and browser cache will live;
whether the run needs network access;
what command proves the first task worked.

2. Make the First Run Evidence-Oriented

A useful first Puppeteer run should produce a small artifact, not just a confident explanation.

Good first-run evidence can be:

a screenshot from a local test page;
a saved HTML excerpt from a controlled URL;
a list of network requests for a single page;
a console log capture;
a short JSON report containing URL, status, title, selector result, and timestamp.

Bad first-run evidence is:

"I inspected the page and it looks fine";
"the automation should work";
"Puppeteer is installed";
a screenshot from a logged-in personal account;
a run that requires production credentials before proving the basic path.

For AI coding agents, the rule should be simple: if no artifact is produced, the browser task is not verified yet.

3. Treat Browser Access as a Permission Boundary

The Doramagic Puppeteer boundary card recommends starting with minimal permissions, a temporary directory, and rollbackable configuration. That is the right default.

Before the agent runs Puppeteer, define the boundary in plain language:

allowed URLs or domains;
whether login state may be used;
whether screenshots may be captured;
whether downloads are allowed;
whether form submission is allowed;
where browser data and temporary files may be written;
what must stop the run immediately.

For example:

Use Puppeteer only against http://localhost:3000.
Do not use existing browser profiles.
Do not submit forms.
Save one screenshot and one JSON report under ./artifacts/browser-smoke/.
If the page redirects outside localhost, stop and report.

This keeps the agent from turning a simple UI check into an open-ended web session.

4. Watch the Real First-Use Pitfalls

The Doramagic Puppeteer pack records source-linked pitfalls around install behavior, browser versions, package alerts, flaky tests, cache behavior, Firefox viewport behavior, and browser binary availability. These should not be exaggerated into "Puppeteer is unsafe" or "Puppeteer is broken." They are check points.

The useful habit is to turn each risk into a verification question:

Which Node version is being used?
Is the project installing puppeteer or puppeteer-core?
Was Chrome downloaded, skipped, or supplied externally?
Does the browser executable exist where the agent thinks it exists?
Is the cache directory temporary and disposable?
Does the smoke test work on the target browser, not just on the agent's assumption?

That is especially important in CI, containers, and remote development environments where browser dependencies differ from a developer laptop.

5. Use GO / HOLD / NO-GO for Agent Runs

For a first Puppeteer capability run, I would use this decision rule:

GO: the task uses a controlled URL, produces a screenshot or report, avoids existing user profiles, and can be rerun.
HOLD: the browser opens, but install path, cache path, executable path, or target URL is unclear.
NO-GO: the agent needs production login state, private data, or external form submission before proving a minimal browser smoke test.

The point is not to make the agent timid. The point is to keep browser automation inspectable.

6. A Safer First Instruction

Instead of asking an AI coding agent to "set up browser automation," start with a smaller instruction:

Using the Puppeteer capability notes, design the smallest safe browser smoke test for this repo.
Use only a local or explicitly approved URL.
Do not use an existing browser profile.
Do not submit forms or use credentials.
Return the planned command, expected artifact, stop conditions, and rollback path before running anything.

That instruction forces the agent to expose the boundary before it touches the browser.

7. What This Helps With

This workflow is useful when you want an AI host to use Puppeteer for screenshots, scraping, browser checks, or UI automation without quietly expanding its permissions.

It does not replace the official Puppeteer documentation. It does not prove production readiness. It does not mean the Puppeteer maintainers endorse this pack.

The useful mental model is:

Puppeteer gives the agent browser hands. Your boundary gives it judgment about where those hands are allowed to go.

Reference: the independent Doramagic Puppeteer project page and manual are here: https://doramagic.ai/en/projects/puppeteer/manual/

Upstream project: https://github.com/puppeteer/puppeteer

Disclosure: this is based on an independent Doramagic capability pack for Puppeteer. It is not affiliated with or endorsed by Puppeteer or Google unless explicitly stated.

Before You Trust an LLM App, Review the Trace

Tang Weigang — Thu, 18 Jun 2026 11:27:03 +0000

When an AI agent fails, the dangerous part is not only the failed output. The dangerous part is the next run.

If nobody reviews the trace, the prompt change, the score signal, and the recovery action, the system can repeat the same mistake with more confidence. That is where Langfuse is useful: it is an open-source LLM engineering platform for observability, metrics, evals, prompt management, playgrounds, and datasets.

The practical question is not "Should I install Langfuse?" The better first question is:

What evidence should I review before trusting the next LLM run?

This is the checklist I would use before adding a Langfuse-oriented workflow to an AI coding host.

One clarification matters: this is not just a prompt and not a prompt library.
The Doramagic Langfuse pack is a capability asset: it includes host
instructions, a prompt preview, a human manual, a pitfall log, a boundary/risk
card, eval checks, a smoke check, a test log, and a feedback path. The point is
to make an AI host reason from source-backed evidence, not to make Langfuse
sound easier than it is.

1. Start With the Failure Review Loop

For an LLM app, a useful review loop needs four things:

A trace that shows what happened.
A scoring or eval signal that says whether the run was acceptable.
A prompt-management habit that records what changed.
A recovery rule that stops the next run from pretending the issue is solved.

Without those four pieces, observability becomes a dashboard you look at after damage is already done.

For agent workflows, the first useful move is simple: require the agent to state what evidence it has before it claims success. If it cannot point to a trace, eval, or smoke check, it should say "not verified" instead of "done".

2. Treat Prompt Changes as Production Changes

Prompt edits often look harmless because they are just text. In practice, a prompt change can alter routing, tool selection, scoring behavior, or output format.

A Langfuse-oriented workflow should make prompt changes reviewable:

What was changed?
Which task or dataset is affected?
Which score changed after the edit?
Was the improvement measured on one example or on a repeatable set?
Can the change be rolled back?

This matters even more when an AI coding agent is involved. If the agent changes code and prompt behavior in the same session, you need a way to separate "the code got better" from "the prompt made the evaluator more forgiving".

3. Do Not Skip the Boundary Card

The Doramagic Langfuse pack marks an important boundary: it is not proof that Langfuse is installed, configured, or production-ready in your environment.

It is also not official Langfuse documentation. Its limits are intentional:
it helps prepare an evidence-review workflow, but it cannot replace upstream
docs, runtime installation evidence, or production security review.

Before real use, check:

Is the test running in a temporary environment or container?
Are production keys, private data, and main config directories excluded?
Can the prompt or config change be rolled back?
Is the run backed by a real trace or only by an agent explanation?
Are failures recorded as evidence, not hidden as "retry noise"?

This is the difference between observability and theater. A dashboard is not a boundary. A trace is not a guarantee. A score is not a release approval unless the team defines how the score is used.

The boundary rule is explicit: if there is no trace, no eval, no smoke check, or
no recorded rollback path, the agent should not claim success. That keeps the
workflow useful even before runtime installation is complete.

4. Watch for Real Integration Pitfalls

The Doramagic pack records source-linked pitfalls that should be checked before first use. Examples include open or version-sensitive issues around scoring behavior, unnamed traces in the UI, Semantic Kernel/Openlit integration behavior, worker shutdown behavior in self-hosted Kubernetes, and idle BullMQ queue timeout behavior.

The important discipline is not to overstate those issues. They are not universal proof that Langfuse is broken. They are reminders to verify the specific version, integration path, and deployment mode you plan to use.

For a first run, I would use a GO / HOLD rule:

GO: a minimal trace is captured, an eval/smoke check is visible, and rollback is clear.
HOLD: the trace exists but scoring, naming, worker behavior, or integration compatibility is unclear.
NO-GO: the agent claims success without runtime evidence, or secrets/production data are required before basic verification.

5. The First Safe Agent Instruction

If you load a Langfuse-oriented capability pack into an AI coding host, the first instruction should not be "set this up in production".

Use a safer first instruction:

Using this pack, identify the first safe verification step for a Langfuse-oriented failure-review workflow.
Do not call external tools unless explicitly approved.
Do not claim Langfuse is installed or working without trace or eval evidence.

The expected result is not a finished integration. The expected result is a boundary-aware next step: what to verify, where evidence should come from, and what would count as failure.

6. What This Helps With

This workflow helps when you are building or operating LLM systems and want an AI assistant to reason from evidence instead of guessing. It is especially useful when the team needs to review traces, evals, prompt changes, and recovery actions before another agent run is trusted.

It does not replace Langfuse's official documentation. It does not prove production readiness. It does not mean upstream maintainers endorse this pack.

The useful mental model is:

Langfuse can give you the observability surface. Your process still needs the boundary, the eval rule, and the rollback habit.

Reference: the independent Doramagic Langfuse project page and manual are here: https://doramagic.ai/en/projects/langfuse/manual/

Upstream project: https://github.com/langfuse/langfuse

Disclosure: this is based on an independent Doramagic capability pack for Langfuse. It is not affiliated with or endorsed by Langfuse unless explicitly stated.

Before You Add More Agents, Design the Control Plane

Tang Weigang — Wed, 27 May 2026 03:09:03 +0000

OpenAI Agents Python makes it easy to describe agents, connect tools, define handoffs, and run agentic workflows. That is useful, but it also creates a trap: teams may start by adding more agents before they define the operational boundaries that make those agents safe to use in a real repository.

The hard part is usually not getting the first demo to run. The hard part is knowing when an agent should start, what it is allowed to touch, what evidence it must leave behind, when it can hand work to another agent, and how the team will recover when the workflow fails.

For production use, I would start with a control plane.

This does not need to be a heavy platform. A Markdown checklist, a JSON policy file, and a trace log can be enough for the first version. The key is that the rules exist outside the model's temporary reasoning. They become part of the workflow, not something you hope the model remembers.

1. Define the task entry contract

An agent should not start from a vague instruction like "fix this feature" or "improve this repo." That may be fine for a toy demo, but it is too wide for real work.

A task entry contract should answer five questions:

What is the goal?
What input is trusted?
What files, services, or systems are in scope?
What is the acceptance standard?
When should the agent stop instead of improvising?

For example, a safer engineering task might say:

Read only the package under src/connectors. Modify only connector_policy.py and related tests. Preserve the git diff. Run the connector test suite. If the requested behavior conflicts with an existing policy rule, stop and return the conflict instead of rewriting the policy.

That kind of instruction is not just prompt polish. It reduces the agent's degrees of freedom. It turns an open-ended request into an executable contract.

The business value is simple: fewer surprising edits, fewer review cycles, and less time spent asking why an agent touched something unrelated.

2. Separate tools by risk

Tool access should not be binary. "The agent can use tools" is too broad. A file search is not the same risk as deleting a directory, publishing an article, or calling a production API.

I prefer three buckets.

Low-risk tools can run directly. Examples: read a file, search for symbols, inspect documentation, list a directory, or open a local artifact.

Medium-risk tools can run if they leave evidence. Examples: modify a draft, generate a patch, run tests, create a report, or produce a migration plan. The output should be inspectable.

High-risk tools require an explicit gate. Examples: destructive git commands, deleting files, pushing to a remote, publishing content, spending money, modifying production infrastructure, or calling external APIs with side effects.

OpenAI Agents Python gives you a framework for building the workflow. It does not automatically know your risk model. That risk model belongs in your engineering system.

If your agent can publish content, the publication action should not be treated the same way as writing a local draft. If your agent can modify code, the modification should not be treated the same way as reading code. If your agent can call production services, the system needs a gate before side effects happen.

This is where many agent workflows become fragile. The model may be capable, but the surrounding system has no authority model.

3. Make handoffs evidence-based

Multi-agent workflows are attractive because they map nicely to human roles: researcher, planner, coder, reviewer, publisher. But every handoff creates a new failure point.

A handoff table should define:

When a handoff is allowed
Which agent receives the task
What evidence must be passed along
Which cases block the handoff

A research agent should not hand work to a writing agent by saying "I found the sources." It should pass source links, key claims, contradictions, uncertain points, and the reason those sources are relevant.

A coding agent should not hand work to a release agent by saying "fixed." It should pass the diff, tests run, tests skipped, remaining risk, and rollback path.

That evidence is the difference between agentic collaboration and a chain of guesses.

The more agents you add, the more important this becomes. Without evidence-based handoffs, every downstream agent has to infer what the upstream agent meant. That makes failures harder to debug and easier to repeat.

4. Treat trace as a product feature

When an agent workflow fails, the least useful conclusion is "the model was unreliable." That may be true, but it does not tell you what to improve.

A useful trace should capture:

The task goal
The input and source material
The rules that were active
The tools that were called
The files or external systems touched
The verification result
The failure reason
The rule or workflow change suggested for next time

You do not need a complex observability backend on day one. A structured Markdown worklog or JSONL trace can be enough. What matters is that failures become training material for the system.

If a failure came from a vague task, improve the task entry contract. If a failure came from excessive permission, tighten the tool policy. If a failure came from a weak handoff, change the handoff table. If a failure came from missing verification, add a test or preflight check.

This is how an agent workflow gets more reliable over time. Not by hoping the next model will magically be better, but by converting failures into rules.

5. Start with one real workflow

The wrong move is to design a giant multi-agent platform first. The better move is to choose one low-risk but real workflow.

Good first workflows include:

Updating documentation after a code change
Reviewing a pull request for missing tests
Classifying issues into actionable buckets
Producing a release note from a verified diff
Preparing a technical article draft with source links and disclosure

For each workflow, define the entry contract, tool policy, handoff table, and trace format. Then run it repeatedly. The goal is not to prove that agents are impressive. The goal is to prove that the workflow reduces repeated coordination while preserving reviewability and rollback.

If a small workflow becomes stable, expand it. If it keeps failing, the trace should tell you whether the problem is the task, the permissions, the handoff, the verification, or the model.

The practical takeaway

OpenAI Agents Python is a useful foundation for building agent workflows. But the production value comes from the control plane around it.

Before adding more agents, define:

How tasks enter the system
Which tools are allowed under which conditions
What evidence is required for handoff
How traces feed back into better rules

That is less exciting than a flashy demo, but it is the difference between an agent that merely runs and an agent workflow that a team can actually trust.

Disclosure: this is an unofficial Doramagic technical note. It is not an official OpenAI publication and does not represent the upstream project unless explicitly stated by that project.

Adopt Codex CLI only after you can explain the source, boundary, review, and rollback model

Tang Weigang — Tue, 26 May 2026 03:51:27 +0000

A lot of teams want to treat Codex CLI as a shortcut: install it, point it at a repository, and hope it saves time immediately. That framing is too shallow for a real codebase.

If you are adopting Codex CLI in a team that cares about quality, the real question is not whether it can write code. The real question is whether the workflow around it is explicit enough to be reviewed, bounded, and reversed. Without those four properties, the tool can create output faster, but it cannot create confidence faster.

1. Start from the source of truth

Before any assistant touches a repository, someone needs to answer a basic question: what is the current source of truth?

That sounds trivial, but it is the first place AI-assisted workflows drift. Teams often test a tool against a repository snapshot, an old issue thread, or a blog post that no longer matches the current implementation. Once that happens, every next step becomes fragile because the assistant is reasoning from stale input.

A useful adoption process starts by checking three things:

the repository or package is the current one
the installation or usage instructions still match reality
the command being run is documented for this version, not an older release

If those checks fail, do not treat the tool as “mostly correct.” Treat the source as unresolved. In practice, a fast assistant reading the wrong upstream source is not a productivity gain. It is a faster way to compound confusion.

2. Make the permission boundary visible

The second boundary is operational scope.

A team should be able to answer, in plain language, what the tool may read, what it may change, what it may execute, and what requires human approval. If those boundaries are hidden in the operator’s head, the workflow is already too loose.

This matters because the early demo of an AI coding tool is misleading. It feels safe when it is only producing text. The risk appears when the same tool is allowed to inspect files, write patches, run shell commands, or touch directories that nobody explicitly intended to expose.

A mature setup does not see permission boundaries as friction. It sees them as the thing that makes the workflow repeatable. The point is not to maximize what the tool can do. The point is to define exactly what it can do so the rest of the team can trust the result.

A practical rule is simple:

read access should be explicit
write access should be narrow
destructive actions should require confirmation
privileged steps should be isolated from exploratory steps

If you cannot describe the boundary clearly, you do not yet have a production workflow.

3. Put review back at the center

The third boundary is review.

This is where many teams get the biggest false win. A tool produces a patch quickly, and the team celebrates the speed. But if the patch is hard to inspect, hard to compare, or hard to reject, the tool has not reduced cost. It has merely moved cost into a later phase when context is already lower.

Review is not a ceremonial step after generation. Review is part of the product.

A good AI-assisted workflow makes the output:

easy to inspect
easy to compare
easy to reject
easy to refine

That means the assistant should be optimized for diffs, not theatrics. If a change cannot be understood in a short review cycle, the workflow is not ready. The best sign of maturity is not that the assistant can generate a large patch. It is that a normal engineer can explain why the patch is acceptable in minutes.

This is also where teams should insist on a clear evidence trail. If a change passes, where is the proof? If it fails, what specifically failed? If the answer is vague, the workflow is too soft to rely on.

4. Treat rollback as part of the design

The fourth boundary is rollback.

Rollback is often treated like cleanup after the fact. That is the wrong mental model. Rollback is part of the design of the workflow itself.

Every real repository will eventually see a bad assumption, an incomplete refactor, a broken command, or a change that looked reasonable until someone reviewed it closely. The question is not whether mistakes will happen. The question is whether recovery is fast enough that the team stays calm.

A rollback-capable workflow has three qualities:

you can identify the last safe state
you can return to it quickly
you can explain what changed without guessing

If those three qualities are not present, then every experiment becomes a one-way door. That is too expensive for a solo team and unacceptable for a shared codebase.

This is the difference between “the tool can help me write code” and “the tool can participate in an engineering system.” The first is a demo. The second is a capability.

5. Use a better adoption question

The wrong question is: can the tool generate good code?

The better question is: can the team trust the workflow around the tool?

That better question breaks down into four operational checks:

Can we identify the source of truth before the tool starts?
Can we define the tool’s authority without ambiguity?
Can we tell whether the change is acceptable in under five minutes?
Can we return to the last known good state without guesswork?

Those are more useful than any demo because they turn a vague technology discussion into a reviewable operating standard.

If any of those questions is “not yet,” the right answer is not to push harder on the model. The right answer is to fix the workflow boundary first.

6. What a real adoption path looks like

For a real team, the best rollout is boring on purpose.

It should begin with a narrow, reversible use case. Not a magical broad permission set. Not an open-ended “let’s see what happens.” A narrow path where the output is easy to inspect and easy to undo.

A good adoption path usually looks like this:

choose one repository
define one class of change
define one reviewer
define one rollback path
measure whether the same standard holds on the second and third run

The repetition matters. The first successful run is easy to overvalue because everybody is paying attention. The real test is the second, third, and tenth run, when the novelty is gone and the tool has to fit ordinary work.

If the workflow does not survive repetition, it is not ready.

7. Why this matters for the team, not just the tool

This approach is bigger than Codex CLI.

Any AI coding tool used in a real repository should be evaluated the same way. The issue is not which vendor is cleverer. The issue is whether the team can maintain control while gaining speed.

When something goes wrong, a mature team should not debate the intelligence of the tool. It should inspect the broken boundary:

was the source stale?
were the permissions too broad?
was the review path unclear?
was rollback not guaranteed?

That framing reduces emotional noise and makes the problem fixable. It also makes the workflow easier to teach to other engineers because the rules are operational, not mystical.

8. The shortest useful conclusion

Codex CLI is worth adopting only when the surrounding workflow is already disciplined enough to keep it honest.

If source is verified, permissions are bounded, review is visible, and rollback is guaranteed, the tool becomes useful. If not, it just helps you create uncertainty faster.

Doramagic project page: https://doramagic.ai/en/projects/codex/
Manual: https://doramagic.ai/en/projects/codex/manual/
Source repository: https://github.com/openai/codex

Non-official note: this is a Doramagic-made, non-official AI capability package. Unless the upstream project states otherwise, it does not represent an official upstream release.

DEV Community: Tang Weigang

Do Not Treat Pydantic AI as an Agent Magic Layer

The real object is a workflow contract

Structured output is not just prettier JSON

Tool calls should start with fake tools

RunContext is a state channel, not a junk drawer

Trace should explain decisions, not dump everything

Toolsets, MCP, and capabilities need an allowlist

A host rule I would load first

A sane first day

Reference roles

Before You Ship an Agent, Make DeepEval Test the Failure Path

Before You Ship an Agent, Make DeepEval Test the Failure Path

Start with failure examples

Keep the first test case boring

Pick metrics by failure type

For agents, trace matters more than the final sentence

Generated goldens still need review

Local eval and cloud sync are different risk levels

A useful host rule

A sane first day

Reference roles

Before an Agent Uses Qdrant, Write the Retrieval Contract

Before an Agent Uses Qdrant, Write the Retrieval Contract

Do not stop at "the vector insert worked"

Treat payload filters as permission boundaries

Vector score is not answer confidence

Quantization and multivectors need a test set

A minimal retrieval contract

A safer first run

Reference roles

Before an Agent Runs Code in E2B, Define the Sandbox Contract First

1. Treat E2B as a sandbox contract, not a permission switch

2. The first run should be disposable and boring

3. Command execution needs a failure contract

4. Do not enable templates, network, volumes, and MCP at the same time

5. Use the pitfall log as a test plan

6. A safe AI-host handoff

7. My working conclusion

Do not treat LangGraph as a longer chain: define state, interrupts, and recovery first

1. The real boundary is State, not the prompt

2. compile() is the boundary between description and runtime

3. Human-in-the-loop is an execution contract

4. Checkpointing is not logging

5. A better first-run checklist

6. My practical take

Do not add LLM evals after launch: use promptfoo to define the failure boundary first

1. An eval is not a score. It is a reproducible judgment set.

2. Agents can run evals, but the scope must be explicit.

3. Many providers do not mean automatic portability.

4. Red teaming should become a negative-case release contract.

A safer first-run path

Before You Let an Agent Convert Documents, Narrow the MarkItDown Boundary

Use it for LLM ingestion, not layout reconstruction

Start with one format, not every optional dependency

PDF and OCR are where expectations drift

Treat the MCP server as a local tool, not a public service

A compact acceptance checklist

Before You Add Memory to an AI Agent, Decide What the Agent Is Allowed to Remember

Before You Add Memory to an AI Agent, Decide What the Agent Is Allowed to Remember

The first useful mental model: three memory tiers

The graph part is not decoration

Backend choice changes the operating boundary

Ontology is the part teams will underestimate

The first safe verification run

The main pitfall

A practical adoption checklist

Before You Give an AI Agent a Browser, Define the Puppeteer Boundary

Before You Give an AI Agent a Browser, Define the Puppeteer Boundary

1. Separate the Two Install Paths

2. Make the First Run Evidence-Oriented

3. Treat Browser Access as a Permission Boundary

4. Watch the Real First-Use Pitfalls

5. Use GO / HOLD / NO-GO for Agent Runs

6. A Safer First Instruction

7. What This Helps With

Before You Trust an LLM App, Review the Trace

1. Start With the Failure Review Loop

2. Treat Prompt Changes as Production Changes

3. Do Not Skip the Boundary Card

`RunContext` is a state channel, not a junk drawer

2. `compile()` is the boundary between description and runtime