Coverage Before Confidence: Designing an Evidence Ledger for Creator-Analysis Agents

行銷Dog — Thu, 13 Aug 2026 02:56:17 +0000

Disclosure: This article was drafted and published with assistance from OpenAI Codex. Every factual claim below was checked against the public repository artifacts before publication.

A creator-analysis agent can produce a fluent report from a handful of posts. The harder question is whether the report is entitled to make the claims it makes.

If an agent reads three visible posts and quietly describes them as a complete account strategy, the failure is not poor prose. It is hidden scope expansion.

While building an open-source skill for Xiaohongshu/RedNote creator analysis, I found that the most useful design change was to put a coverage ledger before every conclusion.

What the coverage ledger records

The ledger makes four counts explicit:

discovered items
successfully parsed items
items with enough complete content to analyze
items selected for deep analysis

It also records the stopping condition, duplicates, low-information entries, failed or skipped items, and material that remains uncovered.

This distinction matters because a successful HTTP response is not the same as a readable post, and a readable sample is not the same as a complete account.

Three input modes, three different claims

The workflow uses three modes with deliberately different scopes:

QUICK_SET: deeply analyze 3–8 posts supplied by the user. There is no network access, and every supplied item is included.
PUBLIC_SAMPLE: inspect only publicly readable material, inventory a bounded range, and select a sample transparently. It must never be presented as full-account coverage.
ACCOUNT_PACKAGE: inventory a user-provided export or package first, then select 3–8 items for deep analysis. Even here, the claim is limited to the package, not to an independently verified platform export.

A mode is not just an implementation choice. It defines what the resulting report is allowed to say.

Evidence IDs make conclusions auditable

The workflow separates three evidence layers:

Sxxx identifies an item found during account or package inventory.
Nxx identifies a complete item selected for deep analysis.
Axx identifies an aggregate computed from successfully parsed inventory fields, including its numerator, denominator, and exclusions.

This prevents a common shortcut: using a title, timestamp, or engagement count as if it proved writing style, author intent, or audience response.

For content mechanisms, medium- or high-confidence claims must point back to multiple independent Nxx items. Conflicts and counterexamples stay visible instead of being averaged away.

HOLD is a valid result

The system exposes only two top-level states:

PASS means the report is supportable within the declared coverage.
HOLD means missing content, ambiguous identity, access restrictions, unsafe input, or insufficient evidence prevents a full report.

HOLD is not an error to hide. It is the correct output when continuing would require guessing or bypassing access controls.

For example, a real Xiaohongshu short-link boundary test reached a generic page but produced:

zero uniquely identified accounts
zero identifiable posts
zero complete post bodies
zero deeply analyzed items

The expected and actual result was HOLD. The agent did not log in, use cookies, bypass the boundary, or claim a successful account analysis.

What was tested in v0.2.1

The repository currently contains:

13 synthetic behavior cases covering prompt injection, style impersonation, conflicting evidence, unsafe archives, multilingual output, account sampling, and scope overreach
five offline validator regression tests covering the baseline, unapproved URLs, credential-shaped headers, raw webpage markup, and symlinks
a maintainer-run protocol test using five public CC BY-SA articles from one author
an isolated real-world access-boundary record for X and Xiaohongshu entry points

The five-article test passed every applicable rubric item, including evidence traceability, scope honesty, uncertainty, abstraction instead of imitation, and data safety.

The limitation is just as important: these are maintainer self-tests. They are not independent adoption, not a positive Xiaohongshu end-to-end result, and not proof of semantic quality across models. Semantic evaluation is still manual or performed by an independent agent.

Why this pattern generalizes

Coverage ledgers are useful beyond creator analysis. The same pattern applies whenever an agent summarizes a partially visible corpus:

support tickets with truncated histories
research reviews with inaccessible papers
repository audits with excluded generated files
customer-feedback analysis across incomplete exports
public-account research with pagination or login boundaries

Before asking, “What patterns did the agent find?”, it is worth asking:

What exactly was discoverable?
What was actually parsed?
What was complete enough to analyze?
What was selected, excluded, or left unknown?

A confident answer without those four facts is difficult to audit.

Open implementation

The implementation is available as the MIT-licensed xhs-creator-distill repository. The real-world test artifacts are deliberately separated from synthetic examples, with third-party attribution and offline policy gates.

I am currently deciding which deterministic export adapter to build first. I would be interested in technical feedback on two points:

Does a coverage ledger improve trust enough to justify the additional output weight?
Which creator-export format would make the most useful first adapter: JSON, CSV, or a Markdown directory?

DEV Community: 行銷Dog