AI-assistance disclosure: I used AI to help draft and edit this article. I checked the technical claims and remain responsible for the final text. I also maintain the open-source repository described here, so this is not an independent review.
Agent capabilities are becoming easier to package and discover. But cataloging a Skill, MCP server, or plugin is not the same as proving that it works for a specific task.
I have been exploring a small open-source project called Agent Catalog Seed. Its purpose is not to create another popularity leaderboard. The experiment is to make missing evidence explicit and versioned.
The state machine matters
For an agent capability, these states are different:
cataloged
-> statically checked
-> installation observed
-> runtime observed
-> task acceptance recorded
Each arrow requires new evidence. A parser accepting SKILL.md does not prove that a host can install it. A tool call returning 200 OK does not prove that the business task is correct. A successful run on one host does not prove compatibility on another.
The catalog should therefore store evidence types rather than flattening them into one score.
A minimal versioned record
The seed currently uses a strict JSON Schema. The example below is simplified, but it shows the separation of concerns:
{
"schema_version": "0.1.0",
"id": "urn:agent-catalog-seed:entry:release-note-checklist-demo",
"kind": "skill",
"installable": false,
"provenance": {
"origin": "ai-assisted-original",
"synthetic": true,
"description": "Synthetic catalog example; not a third-party package"
},
"requested_permissions": {
"filesystem_read": [],
"filesystem_write": [],
"network": false,
"network_destinations": [],
"data_destinations": []
},
"compatibility": [{
"host": "unverified-host",
"status": "unknown",
"evidence": []
}],
"evaluation": {
"status": "not_evaluated",
"independent_review": false,
"evidence": []
},
"conflicts": {
"maintainer_is_author": true,
"disclosure": "Maintainer-authored synthetic example"
}
}
The current seed deliberately makes entries non-installable and unevaluated. It is safer to publish an honest unknown than to convert a roadmap or author claim into a result.
1. Provenance before recommendation
A useful record should identify:
- source URL and maintainer;
- release, revision, or content digest;
- artifacts actually inspected or tested;
- license and redistribution basis;
- relationship between evaluator and author.
The version identity is essential. When a dependency changes, we need to know which evidence is still applicable.
2. Requested permissions are not enforced permissions
The catalog can record filesystem, network, command execution, credential, and data-destination requirements. But it should not pretend that metadata enforces a sandbox.
The host, user, identity provider, and remote service still control authorization. Static documentation can be wrong or incomplete. Runtime observation may reveal behavior, but it remains scoped to the tested environment.
This is why “security” should not be one Boolean field. At minimum, separate provenance and integrity, requested and granted permissions, static findings, runtime observations, and unresolved risks.
3. Task evidence needs an acceptance contract
Before testing, define the scenario and acceptance criteria. Then record:
- host, model, adapter, dependency, and configuration versions;
- input boundary and whether the data is synthetic or authorized;
- planned attempts, retries, human interventions, and timeouts;
- every pass, failure, interruption, and indeterminate result;
- time and cost basis;
- who accepted the output.
Do not remove a failed run because a retry succeeded. Do not change the acceptance criteria after seeing the output without recording the change.
4. Evidence should expire
A test result needs an invalidation policy. A new release, host update, model snapshot, permission change, or task revision may move a record from observed back to retest_required.
This is where version monitoring becomes more than a notification feed. The useful question is not only “what changed?” but “which prior task evidence may no longer apply?”
What the repository proves today
The public seed contains a versioned Schema, a synthetic example, a validator, boundary tests, evaluation and governance templates, and a real public narrated-video-review Skill.
The Skill passed static format validation. It has not been independently tested in a real video edit or verified across multiple hosts. There is no live marketplace, no automatic evaluation service, no independent certification, no real user rating, and no paying customer.
The repository is useful today as an inspectable proposal, not as a trust badge.
A reproducible feedback request
If your team has a recurring failure caused by a Skill, MCP server, plugin, or dependency update, open a GitHub Issue with a non-sensitive test case:
- task and acceptance condition;
- capability and version;
- host or integration context;
- observed failure or maintenance burden.
Do not post secrets, customer data, production credentials, or private files. Synthetic and redacted cases are welcome.
The goal is to learn whether version monitoring, customer-task regression evidence, and team allowlists solve a repeated engineering problem—not to manufacture a large catalog before that problem is validated.
Author disclosure: Leo Liu initiated and maintains Agent Catalog Seed. This article is an AI-assisted, human-reviewed technical note. It is not sponsored and does not represent an independent security assessment.
Top comments (0)