The Most Useful Result in My AI Research Was the One That Failed

Sepehr Bayat — Fri, 14 Aug 2026 13:17:13 +0000

What a broken cohomology claim taught me about measuring coherence in multi-agent systems

I started the Phronesis Index with an idea that felt almost obvious once I saw it:

An intelligent system should not only produce good local answers. Its local commitments should also fit together globally.

That sounds modest. It is not.

In a multi-agent system, every agent can be locally rational, every message can pass validation, and every pairwise relationship can look consistent. Yet the full network can still contain a contradiction that appears only when information is composed around a cycle.

I wanted a single diagnostic signal for that hidden failure mode.

The result I built looked promising. Then the audit broke its central interpretation.

And that failure may have clarified the research more than a clean result would have.

The original intuition

Most AI observability is built around outputs and events: task completion, latency, token cost, confidence, policy violations, exceptions and human ratings.

These metrics are useful, but they mostly observe agents one at a time or workflows one step at a time.

Multi-agent systems add another layer. A network can be locally functional while globally incoherent.

Consider three local models connected in a loop. Each neighboring pair has a valid translation map. If you inspect any one edge, nothing is wrong. But transport a value around the whole loop and it may return changed. The contradiction is not located in one agent. It belongs to the topology of the system.

Cellular sheaves give us a language for this. They represent local data, the restriction maps between contexts, and the conditions under which local sections can be glued into a global one.

The Phronesis Index was my attempt to turn that structure into a practical health signal.

The name comes from the Greek idea of phronesis, or practical wisdom. I am not claiming that wisdom can be reduced to one number. The narrower thought is this: practical judgment requires local decisions to remain compatible when they meet reality—and one another.

The mathematical mistake

The first implementation constructed a degree-0 sheaf Laplacian and counted near-zero eigenvalues beyond an estimate of the dimension of global sections. The manuscript interpreted that excess as a proxy for first cohomology, H^1.

That interpretation was not justified.

For a standard cellular sheaf, the kernel of the degree-0 Laplacian corresponds to H^0: global sections. It does not generally decompose into H^0 + H^1 in the way the estimator assumed.

If the obstruction of interest lives in degree 1, the mathematics has to look in the correct degree—through the degree-1 Hodge Laplacian, the cokernel of the degree-0 coboundary where appropriate, or another construction with a proved relationship to the target.

This was not a cosmetic correction. It removed the strongest theoretical interpretation of the current scalar.

The experiment also refused to cooperate

The safe-RL experiment tested whether adding the Phronesis-based signal to reward shaping would reduce cumulative safety violations.

In the latest reproducible run, it did not.

The Phronesis-shaped variant had a worse mean violation count than the baseline, and the difference was not statistically significant. That means the current evidence does not support the safety-improvement claim.

It is tempting to explain away a negative result: the coefficient was wrong, the environment was too small, the signal was noisy, the seeds were unlucky.

Any of those could be true. None is evidence.

The responsible conclusion is smaller: this experiment did not demonstrate the claimed benefit.

Why I did not abandon the idea

The failed claim and the failed experiment attack a formula. They do not settle the underlying question.

Can we observe when local commitments stop composing into a coherent global state?

I still think this is a useful research direction for three reasons.

First, the failure mode is real. Distributed systems, multi-robot teams, federated decision processes and agentic workflows all face problems that are not reducible to one bad component.

Second, the sheaf-theoretic framing is unusually well matched to the structure of the problem. It does not force heterogeneous local contexts into one uniform representation before asking whether they agree.

Third, observability needs structural signals. As agent networks become more autonomous, output monitoring alone will not tell us whether the internal web of commitments remains mutually possible.

The strongest part of the project may therefore be the question it makes operational, not the first equation proposed to answer it.

The revised research program

I am now separating the work into two tracks.

The first is mathematical: construct an estimator whose relationship to degree-1 obstruction is explicit, reproducible and tested against synthetic sheaves with known ground-truth cohomology.

The second is empirical: treat spectral consistency as a narrower engineering signal and test whether it predicts failures that ordinary graph, residual or constraint metrics miss.

The tracks should meet only after each can stand on its own.

That creates a better standard for the next version:

a correct object of estimation;
known ground truth;
adversarial counterexamples;
baselines that can beat the method;
confidence intervals instead of single runs;
and claims that shrink when the evidence shrinks.

Research is allowed to become smaller

There is a strange incentive in public research to make every revision sound like expansion.

Sometimes progress is the opposite.

You remove the theorem you cannot defend. You rename the metric so it says less. You publish the negative result. You split one grand paper into two answerable questions.

The idea becomes smaller, but the work becomes more real.

The Phronesis Index is currently a research prototype and a preprint under revision. It is not a validated safety metric, and the current implementation should not be treated as a proven estimator of first cohomology.

But I believe the central question remains open and important:

What would it take to make global coherence observable before a multi-agent system fails?

If you work on cellular sheaves, Hodge theory, distributed systems, multi-agent reinforcement learning or AI observability, I would value your criticism.

Repository: https://github.com/sepehrbayat/phronesis-index-nmi

Preprint v1: https://doi.org/10.21203/rs.3.rs-8830176/v1

— Sepehr Bayat