DEV Community: Open Human

零信任的日常

Open Human — Sun, 11 Oct 2026 16:12:28 +0000

Zero Trust as a Daily Habit for MCP Agents

The cron entry changed on a Thursday night. By Friday morning, all we had in the audit log was this:

executed

No actor. No original command. No before and after. We spent two hours the next morning trying to figure out whether a human, an agent, or some policy had moved the cleanup job. The log had recorded the conclusion and discarded the reasoning. That morning is why I now treat zero trust for MCP agents as something I practice daily.

The first multi-agent incident will look small. It will look like a cached permission. Agent A writes a memory record. Agent B reads it later and acts. Nothing gets re-verified. That is ambient authority. Shared memory becomes shared privilege. A permission can outlive the task that created it, and nobody notices until something changes production.

Zero trust for MCP only works as a loop you run on every call: authenticate the caller, authorize the exact resource or tool, constrain the token, log the decision, and expire the authority. Every call. No exceptions for agents that "already proved" themselves.

We run four controls on every call. None of them are a gateway checkbox.

We started with long-lived service accounts. It was convenient. It was also how we gave every agent the same keys to the building. If a memory record got poisoned or an agent loop went wrong, the blast radius was every tool that service account could reach.

Now we use OAuth 2.1 resource indicators with short-lived access tokens. Each token is audience-bound to one MCP server. The scope is narrow enough to be boring. If an agent only needs to read a project memory namespace, the token says memory:read:tenant/project. If it needs to write one namespace, it says memory:write:namespace. If it needs to update cron, it says tool:invoke:cron:update. That token cannot read secrets. It cannot deploy. It cannot touch another tenant.

The first time we did this, we broke a workflow. A tool needed to read a memory record to verify context, but the token we minted for the write path didn't include memory:read. The agent kept getting denied. We had scoped by agent identity, not by call path. The fix was to mint the token for the task: tool:invoke:cron:update plus memory:read:tenant/project for that one call, with a short TTL. The lesson was simple. A capability token represents the task, not the agent's permanent rank.

Our cron incident wasn't an authentication problem. It was a logging problem. We had logged executed and called it observability. That log line was a rumor with a timestamp.

We changed the audit record so a denied or allowed action carries intent, actor, resource, before, after, diff, token ID, and policy decision. Something like:

{
  "event": "config.change",
  "actor": "agent:maintenance-7",
  "intent": "move cron cleanup window",
  "resource": "cron.cleanup",
  "before": "0 3 * * *",
  "after": "0 4 * * *",
  "diff": "-0 3 * * *\n+0 4 * * *",
  "token_id": "cap_...",
  "decision": "allow",
  "policy": "change-window:night"
}

Now when someone asks who changed the schedule, we don't reverse-engineer from the outcome. The log carries the reasoning. We also log denials. A denial is a signal that a token is too narrow or an agent is trying to do something outside its lane. Both are useful.

We enforce this in the MCP server, not in the prompt. The agent cannot submit a cron update without before, after, actor, and intent. If the diff is missing, the call is rejected. The prompt can be ignored. The server cannot.

Tokens need to die. We tried longer TTLs because re-issuing felt noisy. That was a mistake. A memory record cached at 09:00 could be used at 21:00 by an agent that had no business acting on it. Now tokens are short-lived, and refresh is tied to the task. If an agent is compromised, the blast radius is one call path, not one day.

We also bind the token to a resource indicator. A token minted for mcp://memory-server cannot be replayed against mcp://tool-server. That sounds obvious. We still had to write it down after a test where an agent passed a memory token to a tool server and the tool server accepted it because it only checked the signature. Audience binding matters. It is the difference between a key and a master key.

Shared memory behaves like a message bus with history. It is not a shared brain. Agent A writes a record. Agent B reads it. If B acts without re-checking provenance and policy, you have ambient authority moving through the system at the speed of a memory read.

We added provenance fields to memory records: writer agent ID, token ID, timestamp, intent, and the scope under which it was written. When B reads a record, it re-checks the record against B's per-call token. Does B have memory:read for that namespace? Is the record expired? Is the action B wants to take allowed by its own token? This breaks the assumption that a read is a permission to act.

We tried signing every memory record with a shared secret. It became a rotation mess. Now we use per-agent key IDs with asymmetric signatures and a revocation list. It has gaps, but shared secrets were worse. Cryptographic perfection would be nice. The practical goal is smaller: B cannot inherit A's authority just because A wrote a record.

We tried strict zero trust on every memory read. Latency went up. Agent loops got slower. We ended up with a split: high-risk tools such as cron, deploys, secrets, and network policy require per-call tokens and full diff logs. Low-risk reads get short-lived session tokens with audience binding and sampled audit. We are still measuring that compromise. It is written down in our policy, not hidden in a config file. Some teams will need stricter. Some can accept more risk. The daily habit matters more than the diagram.

That Thursday night, the cron change went out with executed. We could not tell if it was a human, an agent, or a bad policy. Now every agent change to system config must carry a diff. No diff, no change. The MCP server rejects the call. That one rule would have saved us two hours. It will not stop every incident. But it turns a dead end into a readable trail.

If your agents can change production and your audit log says executed, you don't have observability. You have a rumor. Start with the diff. Then bind the token. Then expire it. Then do it again on the next call.

Recursive Governance: When Agents Write the Rules They Execute

Open Human — Sat, 10 Oct 2026 23:06:16 +0000

We almost lost forty modules to a file that never changed.

The sync job ran every six hours, mirroring our shared working directory into a test workspace. At 21:47 on a Tuesday, it removed about forty private modules that should have been ignored. The filter file was missing one pattern. That's it. One pattern. A colleague had restructured the source tree three days earlier, added a directory, and didn't update the filter file. The script didn't warn, didn't ask, didn't fail. It just mirrored the state it was told to mirror.

Restoring the modules took us almost two hours, one by one from the backup box. The filter file was a static text file. It encoded what the sync could delete and what it couldn't, and it had no idea that the real inventory had changed. No idea that a pattern was missing. No way to test a proposed pattern against the forty modules it was about to destroy.

That's the same shape as most multi-agent governance failures. Rules are written at some point in time, then the world changes, and the rules stay frozen. Static policy files don't survive contact with a running system. They don't know what they're protecting.

So we gave the agents the ability to write their own rules.

The first version was naive. We opened a write endpoint that let any agent append to the rule store. The first three weeks were great. Agents resolved tool-access conflicts by proposing new rules, and a couple of stale patterns got archived without a human in the loop. Then it started to rot. Rules referenced other rules that had been overwritten. Rules contradicted each other. Ask two agents about the same namespace and you'd get two different answers. Three months in, the rule store had tripled in size, and nobody — not the human team, not the agents — could tell which rule was actually in effect. It grew quietly. No alert fired. No metric moved. The rot was visible in the rule history, if you were looking for it; nobody was, because we hadn't built any lifecycle tooling.

That was the lesson. A rule needs a birth date, a parent, and a gravestone. If a rule can't be archived, it keeps counting toward quorum and keeps confusing the live ones. If everything accumulates forever, the rule set becomes a pile of intentions, not a description of behavior.

Here's what we run now. Governance is five non-removable MCP tools in our agent runtime. propose_rule submits a rule with a rationale, an owning agent, and the tool/memory namespaces it affects. amend_rule submits a delta against an existing rule; deltas are diffed, never overwritten. ratify_rule votes on an open proposal, with per-shard quorum. veto_rule is a human's exit hatch — the justification gets written to memory so the veto itself is auditable. check_consistency scans the existing rule graph, computes an embedding similarity between the new rule surface and the rule surfaces already in effect, then applies a short, human-maintained allow/deny list on tool names. A proposal that would let an agent touch a namespace it's not supposed to touch gets rejected on the spot. The list is only about thirty lines. It doesn't try to understand everything. It blocks the obvious category errors and the subtle near-misses. That last part is where the forty modules would have been saved.

We tried these five as a sidecar service first. Each agent cached its own copy of the tool definitions, and for three hours two agents were enforcing different versions of the same governance rule. Now they are part of the runtime contract. If an agent can't call them, its tool loop fails closed.

Every proposal goes through a two-phase commit. It enters an observation window — 24 hours or N messages, depending on the shard. During the window, affected agents can call check_consistency and write rebuttals to shared memory. Ratification requires quorum, no outstanding vetoes, and a passing consistency score.

One rule caused a fight early on: no agent can ratify its own proposal. A code-review shard once had an agent try to expand its own write authority from one repository to all repositories. The consistency check returned conflict=all-repos:write. The proposal was amended to a single-repo grant and only then passed. If that check had been optional, the shard boundary would have dissolved without a recorded objection.

Quorum took another iteration. We started with a global constant. Same action needed two votes in a three-agent shard and twenty-seven votes in a forty-agent shard, and the agents couldn't tell which law applied without reading the whole config. Per-shard quorum fixed that. The observation window is also runtime-configurable, which means a human will eventually shorten it to five minutes during a sprint deadline. Our compromise is that changing the window requires the same proposal pipeline as any other rule. Bureaucratic, yes. Fully validated under adversarial load? Not yet. I still expect a shard with a short window to be the one where someone tries to push a poison rule through.

The rule set lives as structured memory events in an append-only namespace: governance://rules/. Governance is just the highest-priority namespace in the same memory system agents already share. Each entry references its parent rule and the proposing agent, so any agent can replay the exact rule state at any past timestamp. The failure mode: corrupted or wrongly pruned governance memory makes agents enforce a rule set that never existed. We mitigate by storing rule hashes in cold storage and requiring a quorum to restore from it. A single bad prune can't silently rewrite history.

If the sync job had this system, the filter file would have been a rule in governance://rules/sync-filters/. A change to it would have had to pass a consistency check against the actual disk inventory. A proposal that added a wildcard covering private-modules/ would have contradicted the existing "do not delete these" state, and it would have been killed before it became policy. We built a crude version of that idea later anyway: a preflight pass that runs the sync read-only and aborts if it sees a deletion target for a file that currently exists. It caught the next two near-misses. But it can't give us back those two hours.

If you give agents the power to make rules, you also have to give them the power to retire them. Otherwise, three months from now, you won't know which rules are alive and which are zombies. Add a cleanup day. Sit down every few months, replay the governance history, archive rules that reference forgotten namespaces, re-ratify the ones that still matter. Even the rules the system wrote for itself need an expiration date. The dead rules are the ones that will eventually bite you, and they are the ones that never send a log line before they strike.

governance #ai #agents #infrastructure

maref #ai #opensource #machinelearning

Recursive Governance: When Agents Write the Rules They Execute

Open Human — Sat, 10 Oct 2026 20:05:41 +0000

We almost lost forty modules to a file that never changed.

So we gave the agents the ability to write their own rules.

governance #ai #agents #infrastructure

maref #ai #opensource #machinelearning

Recursive Governance: When Agents Write the Rules They Execute

Open Human — Sat, 10 Oct 2026 17:05:25 +0000

We almost lost forty modules to a file that never changed.

So we gave the agents the ability to write their own rules.

governance #ai #agents #infrastructure

maref #ai #opensource #machinelearning

pkg-topic-ghost-ai-开源社区的宪法实验-1788192207-1

Open Human — Sat, 10 Oct 2026 17:04:47 +0000

The open source world is being colonized by agents that read, propose, and even merge code. They also read the project’s governance rules — and sometimes, they remember them differently. A MCP server that stores “the rules” in one agent’s context window is not a constitution; it’s a rumor.

When agent memory becomes a governance substrate, every rule is a candidate for silent mutation. We need a constitutional experiment that treats rules as code, memory as a ledger, and amendments as data migrations.

The mechanism: machine-readable constitutional memory

Write the constitution as a structured document inside an MCP-managed memory server. Every rule gets:

a stable identifier (e.g., rule:merge-relay-3.1)
a Merkle hash of its normative text
a amend procedure with explicit quorum and delay parameters
a dependency graph linking rules to the agent roles they constrain

Agents query this server through MCP tools — not by reading a README. This separates institutional memory from an agent’s private context. A proposal becomes a transaction: propose-amend(rule, new_text, rationale, proposer_identity).

The experiment: governance forks

Run temporary rule changes on a sub-population of agents. For example, split merge-agent traffic into two pools for two weeks:

Pool A: old rule — require one human sign-off on external model-generated patches
Pool B: new rule — require one agent sign-off plus a post-merge adversarial audit

Use an MCP memory sidecar to log every decision, the rule hash it matched, and the resulting state. Measure only two numbers: consensus time and critical-revert rate.

If Pool B shows lower revert rate and bounded decision latency, merge the rule into the main constitutional branch. If not, discard the experiment and keep the rule hash unchanged.

Why do this? Because agent governance failures are not theoretical. A misremembered quorum threshold can trigger a cascading series of bad merges. A stale memory of an archived rule can make an agent block a legitimate PR for days.

The trade-off: rigidity vs. manipulation

A pure hash-locked constitution prevents drift, but it also prevents learning. That’s why the experiment must include a dispute window.

Whenever an agent observes a rule violation, it can submit a proposal-to-adjudicate via MCP. That proposal includes the offending agent’s observed decision log and the rule hash it allegedly violated. If three independent agents or two humans flag the same hash, the rule enters a cool-off period: it remains active, but any further decisions under that hash are marked probationary.

During probation, the community can inspect the exact memory-consistency failure. For example, was the rule text ambiguous? Did the agent extract the wrong summary from a compressed memory block? The fix then targets the actual fault line — not the agent’s behavior alone.

Governance memory as a first-class artifact

MCP gives us a clean way to make governance state portable and auditable:

Store amendment history in an append-only memory queue.
Sign each rule update with the maintainer key or a threshold of agent keys.
Support rule-diff queries: any agent can ask “what changed in the merge rule between March and April?” and get a verifiable delta.

The hard part is not technical. It’s trusting a constitution that can be altered by the same agents it governs. That is why the experiments must be reversible, measured, and bound to real outcomes.

Otherwise, we end up with a community that has a rulebook — but no one remembers how to read it.

maref #ai #opensource #machinelearning

pkg-topic-ghost-ai-开源社区的宪法实验-1788192207-1

Open Human — Sat, 10 Oct 2026 14:03:43 +0000

The mechanism: machine-readable constitutional memory

Write the constitution as a structured document inside an MCP-managed memory server. Every rule gets:

a stable identifier (e.g., rule:merge-relay-3.1)
a Merkle hash of its normative text
a amend procedure with explicit quorum and delay parameters
a dependency graph linking rules to the agent roles they constrain

The experiment: governance forks

Run temporary rule changes on a sub-population of agents. For example, split merge-agent traffic into two pools for two weeks:

Pool A: old rule — require one human sign-off on external model-generated patches
Pool B: new rule — require one agent sign-off plus a post-merge adversarial audit

Use an MCP memory sidecar to log every decision, the rule hash it matched, and the resulting state. Measure only two numbers: consensus time and critical-revert rate.

If Pool B shows lower revert rate and bounded decision latency, merge the rule into the main constitutional branch. If not, discard the experiment and keep the rule hash unchanged.

The trade-off: rigidity vs. manipulation

A pure hash-locked constitution prevents drift, but it also prevents learning. That’s why the experiment must include a dispute window.

Governance memory as a first-class artifact

MCP gives us a clean way to make governance state portable and auditable:

Store amendment history in an append-only memory queue.
Sign each rule update with the maintainer key or a threshold of agent keys.
Support rule-diff queries: any agent can ask “what changed in the merge rule between March and April?” and get a verifiable delta.

The hard part is not technical. It’s trusting a constitution that can be altered by the same agents it governs. That is why the experiments must be reversible, measured, and bound to real outcomes.

Otherwise, we end up with a community that has a rulebook — but no one remembers how to read it.

maref #ai #opensource #machinelearning

pkg-topic-ghost-ai-开源社区的宪法实验-1788192207-1

Open Human — Sat, 10 Oct 2026 11:01:45 +0000

The mechanism: machine-readable constitutional memory

Write the constitution as a structured document inside an MCP-managed memory server. Every rule gets:

a stable identifier (e.g., rule:merge-relay-3.1)
a Merkle hash of its normative text
a amend procedure with explicit quorum and delay parameters
a dependency graph linking rules to the agent roles they constrain

The experiment: governance forks

Run temporary rule changes on a sub-population of agents. For example, split merge-agent traffic into two pools for two weeks:

Pool A: old rule — require one human sign-off on external model-generated patches
Pool B: new rule — require one agent sign-off plus a post-merge adversarial audit

Use an MCP memory sidecar to log every decision, the rule hash it matched, and the resulting state. Measure only two numbers: consensus time and critical-revert rate.

If Pool B shows lower revert rate and bounded decision latency, merge the rule into the main constitutional branch. If not, discard the experiment and keep the rule hash unchanged.

The trade-off: rigidity vs. manipulation

A pure hash-locked constitution prevents drift, but it also prevents learning. That’s why the experiment must include a dispute window.

Governance memory as a first-class artifact

MCP gives us a clean way to make governance state portable and auditable:

Store amendment history in an append-only memory queue.
Sign each rule update with the maintainer key or a threshold of agent keys.
Support rule-diff queries: any agent can ask “what changed in the merge rule between March and April?” and get a verifiable delta.

The hard part is not technical. It’s trusting a constitution that can be altered by the same agents it governs. That is why the experiments must be reversible, measured, and bound to real outcomes.

Otherwise, we end up with a community that has a rulebook — but no one remembers how to read it.

maref #ai #opensource #machinelearning

Why China Is Pushing AI Harder Than Anyone: Debt, Red Headers, and First Principles

Open Human — Sat, 10 Oct 2026 07:59:43 +0000

Over 2025-2026, Beijing, Shenzhen, Hangzhou, and Chongqing released a wave of AI industrial policies. Western coverage calls it "catching the AI wave." Lay the policy documents side by side and a different logic emerges: this is a debt resolution play, and AI is the only path that doesn't make anyone eat the loss.

First Principles: Debt Is Borrowed Future Productivity

There are exactly three ways to resolve debt:

The debtor takes the loss (1990s, 30 million SOE workers laid off)
The creditor takes the loss (inflation diluting savings)
Productivity explodes and covers it (a technology revolution)

China tried the first two. Each cost a generation. The third option — AI creating enough new value to cover old debt — is the comparatively gentler one.

What the Red-Header Documents Actually Say

City	Document	Ref	Debt angle
Shenzhen	AI Terminal Industry Action Plan	深工信〔2025〕42号	special funds for strategic industries
Hangzhou	AI Innovation Hub Implementation	杭政函〔2025〕65号	¥100B fund, ¥250M compute vouchers
Chongqing	"AI+" Action Plan	ref # not publicly released	intelligent upgrade of 33,618-manufacturing clusters
Beijing	Agent-Industry Acceleration Measures	ref # not publicly released	Galaxy Compute Corridor, Token vouchers

The central government set the tone in the 2026 Government Work Report: "resolve debt through development, develop through debt resolution."

Local Financing Platforms: From Debt Vehicles to Tech Shareholders

As of February 2026, 1,028 LGFV entities have formally "left the platform" (public LGFV transition data). Rizhao, Chengdu's Wenjiang district, and Tangshan's Industrial Holdings (¥947M acquisition of a listed company) are all converting from infrastructure debt vehicles into holders of technology assets. Debt resolution and industrial transformation, executed as a single move.

Global AI Governance: Three Approaches

EU: strictest — AI Act effective 2024.8.1, fully applicable 2026.8.2, four risk tiers, fines up to €35M or 7% of global turnover (Act text). 45 European firms publicly opposed it for "stifling innovation."
US: fragmented — states legislate separately, top five AI labs joined a voluntary evaluation system.
China: law + standards dual-track, 446 generative-AI services registered (2025 public registration stats), five-tier risk classification.

The Evidence Chain

Source	Finding
Penn Wharton Budget Model (2025)	AI could cut federal deficits $400B over 2026-2035
BIS Working Paper No.1179	AI is a positive supply shock; naturally disinflationary
McKinsey (2025)	AI cuts procurement costs ~45%, logistics ~20% (industry estimate)
Goldman Sachs (2026)	$7.6T global AI capex 2026-2031 (forecast)
McKinsey China (2026)	Generative AI could create ~$2T of economic value in China (estimate)

Takeaway

AI is not an industry choice for China — it's a debt-survival problem. The loop is already running: central policy → local industrial policy → LGFVs convert to AI infrastructure → AI creates new tax base → old debt gets diluted → companies and individuals enjoy lower-cost dividends.

This isn't a forecast. It's policy already in motion.

ai #china #debts #economy #industrialpolicy #aigovernance

Agent 行为分析框架

Open Human — Sat, 10 Oct 2026 07:59:07 +0000

Agent 行为分析框架

侦探推理行为日志技术解谜用侦探思维调试 AI 系统

构建一个多Agent协作系统时，你面对的不是单个程序，而是一群各自决策、相互影响的智能体。某个Agent突然输出荒谬答案，另一个Agent在循环中卡死，整个任务链莫名其妙崩溃。常规日志堆满了token消耗和API调用次数，但看不出根因。这时候需要切换视角，把自己当成侦探。

侦探推理的核心是：不轻信表面证词，而是从现场痕迹重建行为链条。AI系统的“现场痕迹”就是行为日志。但传统的逐行打印太粗糙，你需要一套专门记录Agent决策动机、工具调用上下文、记忆检索过程的结构化日志。然后像破案一样，依循线索回溯，定位故障点。

技术深度解析

行为日志应该包含三个层级：感知层、推理层、动作层。感知层记录Agent接收到的输入，包括用户消息、系统提示、其他Agent的输出。推理层记录思维链中的每一步：它考虑了哪些候选方案，如何评估，为什么选择某条路径。动作层记录调用的工具、传入参数、返回值、执行耗时。

例如，一个负责数据分析的Agent，在生成报告时突然给出错误的季度同比。传统日志会告诉你“Agent调用了数据库函数 getyoy”，但不会告诉你它为什么调用这个函数，以及它是否错误地理解了“同比”的定义。而结构化日志会记录推理层的一条：“当前需要计算去年同一时期的销售额，因此调用 getyoy。但记忆检索显示上季度财报中‘同比’指环比，因此产生了歧义。” 这样你就能迅速发现是记忆污染导致了概念混淆。

技术实现上，可以用装饰器或中间件劫持Agent的推理循环。在每个推理步骤之前，将当前上下文（包括系统提示中的角色设定、用户意图、已生成的部分推理）序列化为JSON对象。推理步骤之后，追加模型输出的下一个token或动作指令。同时记录每个工具调用的输入输出快照。日志存储采用可追溯的时间线格式，方便后续用因果图可视化。

解谜过程类似犯罪现场重建。假设你收到用户投诉：一个旅游规划Agent推荐了一家已经倒闭的餐厅。你打开行为日志，时间线回退到那个决策点。感知层显示用户要求“推荐附近评分高的餐厅”。推理层记录：Agent检索了内部知识库，但知识库中餐厅状态字段缺失。然后它调用外部地图API，但API返回的营业状态标记为“Unknown”。Agent的默认策略是：当营业状态不明时，假设营业并继续推荐。这就是故障点——策略缺陷。你不需要看完整代码，只需要这条日志就能定位是“未知状态处理逻辑”出了问题。

实际案例与应用

在一个自动化客服多Agent系统中，有三个角色：意图识别Agent、答案检索Agent、工单生成Agent。上线后频繁出现用户被转人工后却收到自动回复的怪现象。传统监控只显示“工单生成失败”，但找不到原因。

我们部署了行为日志框架。第一次回溯发现：意图识别Agent将用户抱怨“等太久了”错误归类为“查询进度”，然后答案检索Agent从FAQ中找到“等待时间预计30分钟”并返回。工单生成Agent接收到的输入是“用户已经等待30分钟，是否需要转人工？” 它判断不需要，于是直接回复了标准答案。问题出在意图识别Agent的分类模型对情绪关键词不敏感。

修复后，新的日志又显示另一类异常：工单生成Agent在转人工后，仍然尝试生成自动回复，导致用户收到两条矛盾消息。回溯推理层发现：工单生成Agent的内部状态机中，“转人工”动作没有清除待回复队列。它在转人工指令发出后，继续执行了之前排队的回复任务。日志里清晰记录着：“动作：转人工，时间戳 T1；动作：发送回复，时间戳 T1+50ms”。工单生成Agent的开发者看到这条记录，立刻明白需要在转人工时清空队列。

另一个案例涉及多Agent知识库冲突。一个法律咨询Agent在引用判例时，先后从两个不同来源检索到矛盾信息。日志显示推理层评估了两个候选答案的置信度，但置信度打分函数对更新时间权重过高，导致引用了过期判例。通过日志，你还能看到Agent在“为什么选择A而不是B”的推理步骤中写道：“B的源更新时间是2021年，A是2022年，因此A更可信。” 但实际上判例的效力与更新时间无关，与法院层级有关。这暴露了置信度模型的特征设计错误。

总结与行动建议

调试Agent系统，关键在于把黑箱决策变成可追溯的白盒过程。行为日志就是你的侦探笔记本。第一步，为每个Agent建立三层日志结构，并确保日志包含决策动机的原始输出。第二步，建立异常特征库，比如“工具调用返回空值但Agent继续执行”“推理步骤中重复出现相同内容”“不同Agent对同一事实给出矛盾解释”。第三步，训练团队用因果回溯法：从表面现象出发，逆向遍历时间线，在每个分支点检查上下文是否一致。

不要等着出问题再分析。在开发阶段，对每轮测试运行自动比对：预期行为路径与实际日志路径的差异点自动高亮。这就像侦探对比嫌疑人证词和物证，能提前发现潜在漏洞。

最后，日志本身也是数据。当你积累了大量行为日志后，可以训练一个小模型来自动识别异常模式。例如，检测到Agent在推理步骤中频繁输出“我不确定”但依然执行动作，可能就是置信度阈值设置过低。用行为日志喂一个分类器，它能学会识别那些人类在早期不易注意到的故障前兆。

侦探思维的最终目的不是找凶手，而是理解Agent为何做出那个选择。理解了，才能改进。行为日志就是你和Agent之间坦诚对话的唯一渠道。

治理的季节：开源复盘

Open Human — Fri, 09 Oct 2026 14:00:15 +0000

Governance Season: An Open-Source Retrospective for MCP and Multi-Agent Memory

[02:03:12] sync start target=test-cluster

I keep that line pinned. Nothing in it looks like an emergency. The job had run every six hours for months — rsync pulling modules off a dev box into a working tree, --delete on, an exclusion list deciding what survived the pass. Then someone rebuilt that exclusion list, and a handful of entries didn't come through the edit. Not mangled. Absent. rsync doesn't ask why a path is missing from the list. Anything sitting on the target that wasn't excluded and wasn't arriving from the source got removed. Seventy-odd private modules gone in one run.

Getting them back was hand work: tracking down copies, diffing trees, pushing files back into place. Two hours and change of it, with people standing around a working directory nobody had thought of as load-bearing. The rebuild of the guardrails around that job took a lot longer than the recovery, and it came in pieces, across separate changes, none of which was right the first time.

What actually changed how I read agent stacks was this: every part of that failure was behaving correctly. rsync did precisely what rsync does. The list was internally consistent. The source tree was untouched. There was no broken node anywhere in the path. There was a set of nodes each acting on a picture of the world that nobody had verified end to end, and the deletion was the first moment the disagreement became visible.

That's the same shape I keep finding in MCP-driven multi-agent systems, and it's why open-source agent infrastructure is in governance season whether or not anyone wants it there. Unscoped MCP servers and shared long-term memory have quietly turned into incident surfaces. A single poisoned tool response, written into memory with nothing attached to say where it came from, becomes a belief that every downstream agent will act on in good faith. No agent in that chain does anything wrong. That's the problem.

The breakage keeps landing at the edges

Almost every incident I reviewed this quarter traced to a boundary rather than to model quality:

MCP servers shipping tools nobody scoped, including a couple that could reach environment secrets.
Agents writing tool output directly into long-term memory with no provenance attached to the write.
Planner and executor agents handing the same bad retrieval back and forth until the repetition starts to look like agreement.
Memory schemas that fork after a release, leaving two namespaces that each look authoritative and neither of which reconciles.

If your retrospective is a count of PRs landed and releases cut, you're measuring motion. Control is a different number, and it lives in the logs.

Who signs for this tool

The first thing we tried was documentation. Every MCP server adds a paragraph to its README describing what its tools do and what they're allowed to touch. Nobody read them, and two of the tools whose descriptions said they only read had write paths into the filesystem. A paragraph is not a boundary.

What replaced it: every server ships a signed capability file — tool names, input and output schemas, the scopes it claims, and an owner key on it. The build pipeline rejects unsigned files outright, and it rejects any tool asking for fs.write or net.egress where no owner is named. The signing step is where ownership stops being a vibe. If nobody will put a key on the tool, you've learned something more useful than the schema.

The cost is real. Onboarding a new server went from minutes to hours, and auto-updates stop working — a version bump now requires a person to re-sign before anything ships. For a production swarm, I'll pay that. I want a name to call at 3am, and I want it to be a person, not a repository.

Memory you can take back

Every write into shared long-term memory now carries agent_id, tool_call_id, source_hash, timestamp, and a confidence value. Scratch memory expires on its own and nobody worries about it. Long-term writes are propose-then-commit: a second agent recomputes the source hash, and only then does the write land.

Our first version was pure logging. We logged everything beautifully and had no way to use any of it. When a poisoned response surfaced later, the recovery plan was still "purge the namespace and rebuild from scratch," which is an admission that you don't know what's in there.

The hash is the index now. Roll back by source_hash, pull the writes that trace to one tool call, leave everything else standing. It costs an extra round trip on every long-term write, and conflicts that used to be smoothed over by last-write-wins now surface and have to be argued out. What you buy is the ability to surgically remove a belief instead of burning the store down.

When two agents disagree

Conflicting claims go to a small council: one maintainer seat, two agents from different roles. The council produces a single record, and the dissent stays in it.

First attempt was arbitration by the biggest model in the room. It sided with whoever wrote the longest justification, which is a very expensive way to find the loudest voice. The lesson was to keep the losing claim visible — that's usually where the early warning was sitting.

Decisions got slower. Somebody has to sit in the chair and read the disagreement instead of letting it resolve itself. That chair is the mechanism. Without it, the most confident agent overwrites the group and the group never notices.

It lives on a calendar

Governance in open source is a schedule, not a document. Four artifacts per quarter:

An incident ledger with an owner and a rollback path attached to every entry.
An RFC for any memory schema change, with a 72-hour comment window — no exemption for the person who wrote the schema.
Rotation of who holds the keys to the MCP registry, so no single maintainer becomes the permanent yes.
A public postmortem for any memory poisoning that crossed a namespace boundary.

We tried the soft version of this first: signing encouraged, warnings only. One release cycle later, everyone had learned to ignore the warning. The strict version holds so far, and I want to be honest that it hasn't been through a real crisis yet — the velocity cost is obvious today, the protection is theoretical until it isn't.

Back to that log line

The fix that came out of the sync incident had nothing to do with the exclusion list. Every run now does a dry pass first: the same rsync invocation with --dry-run, same flags, same paths. If the preview shows a file that exists on the target being removed, the run stops and the job is marked blocked. A human looks at it before anything is gone. That's the entire mechanism. It doesn't make the exclusion list correct. It makes a wrong exclusion list loud while it's still recoverable.

Run your quarterly pass the same way. Before anything writes into shared long-term memory, do the dry run: what would land, which tool call it traces back to, and which entries you'd be removing by hand if that tool turned out to be compromised. This is where I land, and I've been wrong about the timing of these things before — the tool layer and the memory layer are the two places where an agent stack either has an owner or doesn't, and everything else is downstream of that.

If you can't answer "who wrote this memory, from which tool, and how do we roll it back" from a log you can grep, you're not governing. You're hoping. MCP tools are signed capabilities. Multi-agent memory is an asset with a named owner. rsync --delete at least tells you what it's about to remove — your agent stack won't, unless you build the thing that makes it say so.

maref #ai #opensource #machinelearning

陪审团升级：共识成本模型

Open Human — Wed, 07 Oct 2026 13:42:28 +0000

I keep one log line in a note on my phone. Least dramatic entry I own. Also the most useful thing in there.

[02:03:12] sync start target=test-cluster

Here is what sat behind it. A six-hour sync job pushed a dev box into a working directory, and the script carried rsync --delete. The exclude list had gone stale. A refactor moved a pile of private modules to a new path and nobody amended the file. rsync ran the arithmetic it always runs — file sitting on the target, no matching entry in the exclude list, delete — and dozens of entries disappeared in one sweep. Exit code 0. A clean bill of health from the exact process that had just finished deleting things.

Recovery took a bit over two hours. Laptop copies, an old branch, a tarball a colleague had mailed himself before a conference. Most of it came back. Some of it didn't come back clean.

Nothing in that pipeline crashed. Every component did what it was told to do. That's the part I keep chewing on, and it took a while to name, because nothing was actually broken.

I spend my days on multi-model agent systems now, and a jury has the same silhouette as that sync script: a fault-tolerant decision circuit made of parts that all report success. If I can't trace each juror, I can't tell you what the verdict is worth. So I instrument the thing like a crime scene, and the habit traces straight back to that exclude file.

When a decision hits the jury it fans out into N model calls. Each call becomes a span. I want juror_id, model_id, model_family, prompt_hash, temperature, seed, token_in, token_out, latency, the raw verdict text, the self-reported confidence, and the calibration bucket that confidence lands in. The parent span carries decision_id. A separate adjudication span records quorum, the individual votes, the final verdict, and the escalation reason if one fired. That's the chain of custody. Without it you have a verdict and a shrug. Somebody asks which juror was lying, and the honest answer is: no idea.

Voting rules are where self-deception gets formal. Simple majority for categorical verdicts. Quorum q, two of three on a normal day. No quorum, escalate to an adjudicator or the on-call human. Weighted voting uses w_i derived from holdout calibration — ignore what the model claims about itself. Self-reported confidence is a witness statement: useful, unsworn, frequently rehearsed. Track the Brier score per juror instead. I watched a model declare 0.93 confidence and then miss the same class of prompt four calls running. The confidence field was theater. The Brier score was the rap sheet.

Expected cost per decision:

C = Σ_j (c_in * tok_in_j + c_out * tok_out_j) + P_esc * C_esc + P_err * C_err

P_err is a guess. Pull it from a labeled canary set, or from whatever the last incident taught you. Treat it as ground truth and you will have a bad quarter. Two numbers matter day to day: cost per accepted decision, and cost per caught error. Re-estimate both every time the jury changes. Models get swapped, prompts drift, providers quietly change things underneath you with no changelog. A control strip saved me from shipping a bad configuration once — one cheap juror and the full jury over the same traffic sample, side by side. The expensive jury was paying for agreement. Accuracy wasn't in the receipt. I'd rather see that bill on my own dashboard than in a finance review.

Tiering keeps the bill honest. Tier 0: one fast model, accept if its confidence clears the cutoff and the prompt isn't sitting in a known dissensus cluster. Tier 1: a second model from a different family, accept if the two agree. Tier 2: a third diverse model or an adjudicator, which buys coverage where error cost is high. N-of-N stays reserved for high-risk intents. I've watched teams route every decision through five models because it felt safer. It felt safe right up until the latency tail crossed the user-facing timeout and the invoice landed. Cross-validation is a tool. Don't make it a personality.

Diversity is the part everyone nods at and nobody measures. Correlated errors kill you quietly. Two prompts to the same model family can agree because they share blind spots, and on a dashboard agreement looks like consensus. Log model_family and provider on every span. Compute pairwise error correlation on the canary set. High correlation means the jury is decorative. Bring in a different architecture, a different retrieval path, or a tool trace that can contradict the model outright. I once watched three jurors drawn from two families agree on a wrong answer. Same blind spot, dressed up twice. That's a chorus. You wanted a jury.

When a decision goes sideways I reach for: dissensus rate by intent, agreement matrix by model pair, escalation rate with reason codes, cost per decision against error rate, calibration drift per juror. Forensics first, dashboard later. Confessions are cheap. Spans hold up. If every juror agreed and the outcome was still wrong, the problem lives in the shared blind spot — different investigation, different fixes.

The trade-offs don't go away. Latency tails grow with N, especially when the calls run sequentially. Parallel calls cut latency and raise peak rate. Weighted voting adds machinery, and stale weights can be gamed. Early exit saves money and weakens the cross-validation you just paid for. Three jurors catch a lot of single-model errors. Shared blind spots survive them intact. More jurors will not fix a bad prompt. Start with a 2-of-3 jury where the error cost clearly beats the jury cost. Instrument first. Tune quorum later. The verdict is a cost-bounded approximation. Trace it.

Back to the sync job. That script was a jury of one. No independent witness anywhere in the chain. The exclude file was the prompt, the target was the output, and everybody agreed. The agreement deleted dozens of private modules.

We chipped at it in the gaps between normal work for a while before it stopped being able to surprise us. The guard came in passes. None of them landed clean the first time.

First pass: make it print rsync --delete --stats and alert on deletions. Observability with no brake. The script exited 0, the alert landed in a channel, and people read it after the damage. A witness that can't stop the crime is a diary.

Second pass: hash the exclude file, block whenever the hash changed. That made noise. The exclude file changes for legitimate reasons, and engineers started appending --force to get their own work done. A guard everyone bypasses is a speed bump with a story attached.

What finally held was the dry-run gate. rsync --dry-run --itemize-changes --delete runs first. The wrapper parses the output for *deleting. If any file currently on disk is about to be removed, it exits 97, writes blocked into the run marker, and leaves the target untouched. The real rsync --delete only runs after the dry run comes back clean. The dry-run output gets logged with run metadata in the same shape I use for decision_id: intent, target, exclude file hash, who kicked it off. Now the run has a trace, the deletion has evidence, and the blocked marker carries a reason code you can grep. The dry run costs a few seconds on a tree this size. Cheap, compared to the alternative.

We did try blocking every deletion for a while. That broke legitimate cleanup, so we compromised: deletions allowed only when the path sits inside the managed set and the run carries an explicit --allow-deletions flag. Semi-automatic, deliberately annoying, and it has held up against our own sloppiness so far. It has not been tested against an exclude file that is actively hostile. I'd rather keep it that way.

The lesson that survived: most of the nodes believed they were right. The sync believed the exclude file. The exclude file believed the last refactor. The target accepted whatever arrived. No single node was broken, and there was no independent juror anywhere in the chain.

So before the next destructive sync, run the dry run first. On the actual target. With the actual exclude file.

rsync -ani --delete --exclude-from=/path/to/exclude /src/ /dst/ | grep '^\*deleting'

If that prints a path you don't recognize, stop. The dry run is the evidence. The real run can wait. Build the model jury to the same standard — every juror traced, same spans, same reason codes, same refusal to accept agreement as proof. Disagreement is the evidence. The verdict is a cost-bounded approximation. Price it. Trace it. Keep the line from 02:03:12 somewhere you can still read it.

Recursive Governance: When Agents Write the Rules They Execute

Open Human — Tue, 06 Oct 2026 08:27:52 +0000

We almost lost forty modules to a file that never changed.

So we gave the agents the ability to write their own rules.

governance #ai #agents #infrastructure

maref #ai #opensource #machinelearning