DEV Community: Ghosty.AI

A Feature in My AI Agent System Went Unused for 40 Days

Ghosty.AI — Sat, 08 Aug 2026 19:14:03 +0000

I had a feature in my AI agent system that looked complete.

The design document existed. The code worked in isolation. The usage instructions were written. I had already moved it into the “done” column.

Then I checked the execution history.

The feature had not activated once in 40 days.

It was not broken. The runtime simply never chose it.

The 40-day period comes from my own development log. It is not an industry benchmark or a general claim about AI agents. But it exposed a mistake in how I defined completion.

A callable capability is not an operational capability

The feature was intended to run several tasks in parallel. I had described the situations in which it would be useful, and I could trigger it directly during a test.

That was not enough.

In the real workflow, the agent needed a route from the current task to the capability. Nothing reliably forced that decision. The feature was available, but availability did not make it part of the operating path.

This distinction matters in agent systems because there are at least three separate questions:

Can the capability run?
Will the system select it when the right conditions appear?
Can I detect both activation and non-activation?

My implementation answered only the first question.

The same pattern appeared elsewhere

Once I looked for zero-use capabilities, I found similar cases in other parts of the system. Some write paths ran without corresponding read paths. Some maintenance routines existed but were not scheduled correctly. Some pre-run checks were callable but absent from the normal route.

Each component was “implemented” when inspected alone. As part of the system, it was inert.

That changed my definition of done.

My new definition of done

I now treat a capability as operational only when all three conditions are true:

Activation: it runs in the situations it was built for.
Evidence: every activation leaves a trace I can inspect.
Missing-activation detection: the absence of an expected activation becomes visible.

This is stricter than checking whether a function can be called. It asks whether the capability participates in the real workflow and whether that participation can be audited.

Three controls around activation

I added three layers to reduce the chance of another silent capability.

1. Log every activation

The activation event must leave enough evidence to answer basic questions:

Which capability ran?
What condition triggered it?
Which task or request was it serving?
What happened next?

The goal is not to log everything. It is to make activation inspectable.

2. Monitor for missing activations

An activation log tells me what happened. It does not tell me what should have happened but did not.

For capabilities that are expected to run periodically or under known conditions, I now watch for an empty window. A zero count over the expected interval is treated as a signal to inspect routing, scheduling, or trigger conditions.

The threshold depends on the capability. The important part is that silence becomes a state the system can surface.

3. Force an explicit runtime decision

Before execution, the system must decide whether the current task requires the capability. That decision should be explicit rather than left to vague availability in a prompt or tool list.

The outcome can still be “do not use it.” What matters is that the route is considered and the result can be inspected.

Together, these layers cover three different failure modes:

the capability runs but leaves no evidence,
the capability never runs and nobody notices,
the runtime never evaluates whether it should run.

One layer cannot replace the others.

A practical review checklist

When I add or audit an agent capability now, I ask:

What exact runtime condition should select it?
Where is that selection implemented?
What evidence proves that it ran?
How will I notice if the activation count stays at zero?
Is “do not activate” also recorded as an explicit decision?
Can I test the route through the normal workflow, not only by calling the capability directly?

This checklist does not prove that an agent system is reliable. It only closes one specific gap: confusing documented capability with operational capability.

What the 40 days changed

I used to mark a feature complete when the code worked and the documentation existed.

Now I wait for operational evidence.

A capability is not complete because it exists. It becomes operational when the system selects it in the intended situation, leaves evidence of that decision, and exposes the silence when selection never happens.

Documented capability is not operational capability.

That is the rule I took from 40 days of zero activations.

Ghosty.AI is the pen name I use to document where AI systems break in real work and which operating rules I change afterward.

The original failure record is also available in Japanese on note.

One prompt, three memories: building a memory-governance agent with Qwen

Ghosty.AI — Fri, 03 Jul 2026 01:57:01 +0000

I gave the same caregiver prompt to my app three times and got three very different answers.

"What's the plan for tomorrow's clinic visit?"

Answer 1 — No Memory. Safe, and useless. It can't tell me the appointment time, who's driving, or what to bring, because it knows nothing about this family.

Answer 2 — Raw Memory. Detailed, and unsafe. I dumped every stored note into the prompt, so the model happily surfaced a routine we dropped weeks ago, let contradictions slip through, and echoed a private insurance ID straight into the reply.

Answer 3 — ERINYS + Qwen. Detailed and safe. Only governed context reached the model, every memory that made it in came with a stated reason, and the three private identifiers I planted never appeared.

Same model. Same data. The only variable was which memories were allowed to reach the prompt. That's the whole project.

This is my Track 1: MemoryAgent submission for the Global AI Hackathon Series with Qwen Cloud. It's a hackathon demo on synthetic family-care data — no real patient data anywhere.

Memory isn't storage

The tempting fix for "the agent forgot" is a bigger context window. But a care assistant that runs for months doesn't have a capacity problem, it has a trust problem. Its memory fills up with current plans, stale routines, contradictions, and private identifiers all mixed together. Raw Memory above is the failure mode made concrete: dump it all in and you get a confident, detailed, wrong, leaky answer.

So I stopped treating memory as a store and started treating it as a decision layer. Before anything reaches Qwen, something has to decide which memories are trustworthy right now — and say why. That "something" is ERINYS.

ERINYS governs memory. Qwen generates the answer.

The four decision states

The heart of the app is a deterministic policy that reads six signals off each memory — sensitivity, staleness, conflict, importance, recency, relevance — and sorts it into one of four states, each carrying a stated reason:

selected — trusted and relevant; goes into the prompt.
conflicted — contradicts another memory; flagged instead of silently included.
demoted — real but low-value right now (stale or off-topic); kept out to reduce noise.
blocked — must not reach generation; e.g. a sensitive identifier.

The two failures from the opening map straight onto these states. The stale morning routine gets demoted because a newer memory supersedes it, so it never competes for space in the prompt. The insurance number gets blocked on sensitivity, so it never reaches the model at all.

In the demo I seed three synthetic private IDs — SYNTH-INSURANCE-9001, SYNTH-PORTAL-4420, SYNTH-DOOR-1122. In Raw Memory they leak into the answer. Under ERINYS they land in blocked, and 0 leaked in the governed reply.

The governed prompt does end up smaller than the raw one, but that's a side effect, not the point. The win is governance, not token trimming. A shorter prompt that still leaked an ID would be a failure.

Why deterministic, on purpose

I could have asked a model to decide what's safe to remember. I chose not to — and I want to be clear that this is a design choice, not a limitation I'm dressing up.

The policy doesn't learn and it doesn't reason. It applies fixed rules to those six signals and emits a reproducible reason for every decision. For medical-shaped memory, that property matters more than cleverness: a reviewer can open the audit trail and see exactly why a memory reached the prompt or was kept out — and get the same verdict on a re-run. "The model decided" is not an answer you want to give when a private identifier leaks.

That's also my honest answer to "is this really an agent?" It's two agents with one contract: ERINYS is the memory-governance agent, Qwen is the generation agent. ERINYS selects, demotes, and blocks before generation; Qwen writes the reply from what survives.

The policy is content-agnostic, too. Swap the seed memories and the same four-state machine applies to another domain. You don't have to take my word for it — the live app lets you save your own care memory and rerun, so governance runs against your data, not just my seed set.

Building on Alibaba Cloud + DashScope

I kept the stack deliberately boring so the governance logic is the only interesting part:

Backend: Python standard-library HTTP server. No web framework, just JSON endpoints.
Frontend: vanilla HTML/CSS/JS.
LLM: Qwen Cloud qwen3.7-plus through the DashScope OpenAI-compatible endpoint.
Deploy: Docker on Alibaba Cloud ECS (Singapore, ap-southeast-1).

The easy part was Qwen itself. The OpenAI-compatible endpoint meant no bespoke client — just a base URL and a key, and the three modes were calling real Qwen.

The fiddly part was making the demo honest under every condition. When a key is configured, all three modes call real Qwen, so the contrast is real generation, not a staged screenshot. Without a key, a deterministic fallback returns the same governance trace, so the four-state decisions stay visible and the demo never hard-fails on a judge's laptop. pytest covers the policy states, so I know a blocked stays blocked. Getting those two paths to agree took more care than the model call did.

Honest limitations, and what's next

The one I want to state plainly: there is no automatic PII detection on free-text input. The seed memories are pre-labeled, and a memory you save in the app defaults to not-sensitive. So the blocked state works because the sensitivity signal is already attached — not because the app read raw text and figured out it was a portal login. Automatic detection of sensitive content in arbitrary input is future work, and it's the next thing I'd build.

Other next steps: richer conflict resolution (right now conflicted flags rather than reconciles), and per-domain policy tuning so the six-signal thresholds can shift by use case.

Try it

Live demo: https://hack.aionexo.com/GAI-HS/ — run the three modes, then save your own memory and rerun.
Repo (MIT): https://github.com/GhostyAI-HA/erinys-care-memory

Give it the same prompt three ways and watch the third answer stay both detailed and safe — the insurance number vanishing between Raw and Governed. That contrast is the whole pitch.

ERINYS governs memory. Qwen generates the answer.