A Digest Quality Score That Knows What It Can't Prove

lakshya agrawal — Tue, 27 Oct 2026 09:51:09 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge: Week 4.

A digest quality score that knows what it can't prove

An AI-generated email digest can sound certain even when a summary, action, or deadline is not supported by the source. GistMyMail's v1 quality evaluator makes several checks explicit and repeatable. It is deterministic and does not ask an LLM to grade its own output. The score is diagnostic; it is not a guarantee of truth.

Project timeline and Week 4 scope

GistMyMail's first repository commit is dated October 25, before Week 4 began on October 26. This is an existing project, not a new project started for this challenge. The work highlighted here is the digest-quality evaluator: a versioned rubric, deterministic source checks, per-item warnings, and a report attached to digest generation.

The challenge asks for brand-new projects. This existing-project contribution may not meet that eligibility requirement, so I am stating the timeline plainly rather than implying that the whole product began during Week 4.

The evaluator changes described below are currently in local working-tree changes and are not yet on the public GitHub branch. The repo link is included for project context; judges will need a checkout that contains the evaluator changes to inspect this feature.

What the evaluator measures

The initial weights are configurable design assumptions, not empirically calibrated values:

Q = 0.25C + 0.30F + 0.15A + 0.15D + 0.15I

Here, C is source coverage, F is factual grounding, A is actionability, D is duplicate content, and I is information completeness. When a component is N/A or unavailable, its weight is excluded and the remaining applicable weights are renormalized. The report records the rubric version, component scores, counts, statuses, warnings, and source references.

Source coverage — 25%

Coverage is the number of eligible account/message pairs represented in at least one digest item divided by the eligible pairs fetched for that digest. Pairs are captured before content deduplication, so two messages with identical content can both count even if one digest item is retained. Each pair has equal weight because the project has no reliable source-importance classification. The denominator covers only the bounded inbox query, not every message in a mailbox.

Factual grounding — 30%

Each non-empty title, summary, action, and deadline is evaluated as a claim. A field is supported when its lowercase, whitespace-normalized text appears literally in a mapped email's subject, body, or snippet. A source ID alone is not evidence.

This rule can miss correct paraphrases and can match words in a negated or qualified sentence. “Unsupported” means no literal match was found; it does not mean the claim is false. A literal match is not proof of semantic entailment.

Actionability — 15%

The denominator includes items in action-oriented categories and any item with an action. An applicable item passes when it has both a title and an action. This checks the item's shape, not whether the action is useful or appropriate.

Duplicate content — 15%

The evaluator lowercases and collapses whitespace in the title, summary, action, and deadline fields. Items are duplicates only when all four normalized fields match; the lowest item ID is retained deterministically. This catches exact normalized duplicates, not semantic repetition.

Information completeness — 15%

The evaluator checks required fields including title, summary, account label, sender, received time, source reference, Gmail URL, and evidence quote. Action and deadline fields apply when the item requires them. Required fields must be present and supported; unresolved source checks are excluded and make the metric partial.

A metric with no applicable items is N/A, not 100. An empty digest with eligible messages gets zero coverage and a warning. Evaluation failures produce an unavailable report, never a perfect score. Synthetic demo data is marked partial.

How it fits the digest lifecycle

The scorer runs after digest generation using the messages and evidence already fetched. It does not call an LLM, read new mail, or send email. The report is persisted with the digest and returned by the existing digest API. Historical digests can still exist without a quality report.

The dashboard currently shows the digest but does not render the quality breakdown. To inspect the report in this build, open browser Developer Tools, choose Network, generate a digest, and inspect the JSON response from POST /api/digest/generate at digest.qualityReport. The report's warnings and source references let a reviewer see why a claim was marked unsupported or unresolved.

Demo path

There is no hosted demo link in the repository documentation today. The documented hands-on demo uses synthetic inboxes and does not require Gmail OAuth credentials or an AI API key:

git clone https://github.com/SecuVortex/GistMyMail.git
cd GistMyMail
pnpm install
pnpm dev

Then:

Open http://localhost:5173.
Click Try Demo Mode.
Click Generate Today's Digest.
In Developer Tools → Network, inspect the POST /api/digest/generate response and expand digest.qualityReport.

The demo report is marked partial because the inbox messages are synthetic. Do not use private emails in screenshots or recordings. These steps require a checkout containing the quality-evaluator changes described above; the public GitHub branch does not include those local changes yet. The setup guide documents the existing demo flow.

Why open innovation matters here

GistMyMail supports open-weight models through OpenRouter and local inference through Ollama. The quick demo uses deterministic mock data so a judge can try the flow without sending email content to an external model. The quality evaluator itself is model-agnostic: its checks remain the same if the generation model changes.

Keeping the rubric and its limitations inspectable lets contributors challenge the weights, test difficult cases, and compare future versions with the same deterministic rules. Local inference can keep email content on the user's machine. The project does not require one hosted model provider for the scoring feature to work.

The “ghost in the machine” for this project is a plausible-sounding commitment that the source email does not support. The evaluator can expose some gaps, but it cannot decide whether a claim is semantically true.

Help make GistMyMail more useful

Useful contributions include:

Add deterministic tests for negation, quoted text, paraphrases, and ambiguous deadlines.
Review multi-inbox coverage, missing evidence, and excluded metrics.
Build the dashboard breakdown and source-linked warning view.
Improve setup, onboarding, and explanations of what the score does and does not prove.

Start at the GistMyMail GitHub repository, open an issue, or send a pull request. For examples involving email, use synthetic or carefully redacted text—never publish private messages, addresses, or tokens.

hello everyone

lakshya agrawal — Sun, 04 Oct 2026 12:43:19 +0000

new to the dev community

DEV Community: lakshya agrawal