How to Budget Python News Monitoring — Evaluation Sets for Retention

Short answer: build the evaluation set around time, repetition, corrections, and missing evidence, then choose a retrieval architecture by the cost of the smallest retained corpus that still passes those tests. For a news monitoring service feeding an internal knowledge-base bot, I would measure that corpus before debating vector databases, embedding models, or rerankers. Index cost at scale is mostly a question about what enters the index, how many representations it creates, and how long each representation remains searchable.

That order matters. A better scoring component cannot rescue an evaluation set that quietly rewards keeping every syndicated copy forever.

Measure first.

Start with the bill, not the retriever

The useful cost model has separate terms for ingestion, retained index state, queries, and evaluation. Treating all four as one “search cost” number hides the lever that a retention policy can actually move. For one planning interval, I use this deliberately plain model:

from dataclasses import dataclass


@dataclass(frozen=True)
class Workload:
    documents_per_day: int
    duplicate_fraction: float
    chunks_per_document: float
    retained_days: int
    bytes_per_chunk: int


def retained_index_bytes(workload: Workload) -> int:
    unique_documents = (
        workload.documents_per_day
        * (1.0 - workload.duplicate_fraction)
        * workload.retained_days
    )
    return round(
        unique_documents
        * workload.chunks_per_document
        * workload.bytes_per_chunk
    )


baseline = Workload(
    documents_per_day=120_000,
    duplicate_fraction=0.35,
    chunks_per_document=3.0,
    retained_days=90,
    bytes_per_chunk=6_144,
)

print(f"retained GiB: {retained_index_bytes(baseline) / 2**30:.1f}")

Those inputs are an example, not a benchmark. Replace every one of them with measurements from the service: accepted feed items per day, the duplicate fraction after canonicalization, emitted chunks per accepted item, actual index bytes divided by indexed chunks, and the retention window. The model is intentionally insensitive to a vendor's unit names. It exposes the dominant term in a particular deployment instead.

The first comparison should change one variable at a time. Cutting retention from 90 days to 30 days divides retained state by three in this model; improving duplicate detection from 35% to 55% reduces both indexing work and retained state; shrinking a representation may reduce retained bytes without reducing document count. None of those changes is automatically correct. The evaluation set has to reveal what search quality each change gives up.

Change	Cost term moved	Likely failure to test	Evidence needed
Shorter hot retention	Retained chunks	An older event disappears from follow-up answers	Age-bucket recall
Collapse syndicated copies	Accepted documents and chunks	A materially revised copy is mistaken for a duplicate	Correction and update cases
Fewer chunks per article	Index writes and retained chunks	A small but decisive passage is no longer retrieved	Passage-level relevance labels
Archive old raw text outside search	Searchable state	Rebuilding or investigating a historical answer takes longer	Restore-time drill

This is where storage skepticism helps: durability of the archive and searchability of the hot index are different requirements. Conflating them makes the expensive representation live as long as the authoritative object, even when nobody has shown that the bot needs both.

How should evaluation sets guide retrieval architecture for a news monitoring service?

An evaluation set should preserve the time-dependent ambiguities that the production system must resolve. A random sample of questions and relevant documents is too forgiving because news is repetitive, corrections arrive after the first publication, names collide, and a correct answer can be “the retained evidence does not support this.” The RAG paper establishes the basic pattern of combining parametric generation with retrieved non-parametric memory; it does not choose a retention policy for a news operation. That decision belongs in the service's own tests.

I would make the unit of evaluation a query at a declared observation time. Each case records an allowed document set, required evidence identifiers, explicitly confusing identifiers, and an answerability label. Observation time prevents a later correction from leaking into an earlier test. Stable content hashes and source-assigned identifiers let the test distinguish a syndicated duplicate from a subsequent article about the same event.

Use at least these slices, sized according to actual traffic rather than an arbitrary equal split:

1. Fresh-event queries where the relevant item is minutes or hours old.
2. Follow-up queries whose evidence crosses the proposed hot-retention boundary.
3. Syndication clusters containing near-identical text from several feeds.
4. Corrections where a later version changes a fact that affects the answer.
5. Entity collisions, such as two projects or companies sharing a short name.
6. Unanswerable queries for which no retained source supports a response.

Don't report only one aggregate score. A large fresh-event slice can conceal total failure on 31-day-old evidence, precisely the failure introduced by a 30-day retention rule. Report recall at a fixed cutoff for every age and failure-mode slice, plus the number of retrieved chunks, index bytes, and indexing operations for the candidate corpus. The quality measure and the resource measure must describe the same frozen snapshot.

No averages alone.

A status code can be part of the fixture without becoming a retrieval label. For example, an ingestion record marked 429 should be retried by the ingestion system and should not be counted as a relevant document that retrieval somehow missed. Mixing pipeline availability with retrieval relevance produces a metric nobody can act on.

Make the retention experiment executable in Python

The following evaluator is small enough to run in a pull request. It does not pretend that token overlap is a production retriever; ranked_ids comes from whichever candidate architecture is under test. The code enforces the parts that teams commonly leave implicit: observation time, required evidence, forbidden leakage, and per-slice reporting.

from collections import defaultdict
from dataclasses import dataclass
from datetime import datetime, timezone
from typing import Iterable


@dataclass(frozen=True)
class EvaluationCase:
    case_id: str
    query: str
    observed_at: datetime
    slice_name: str
    required_ids: frozenset[str]
    forbidden_ids: frozenset[str]
    answerable: bool


@dataclass(frozen=True)
class RetrievalResult:
    case_id: str
    ranked_ids: tuple[str, ...]


def recall_at_k(case: EvaluationCase, result: RetrievalResult, k: int) -> float:
    if not case.required_ids:
        return 1.0
    retrieved = set(result.ranked_ids[:k])
    return len(retrieved & case.required_ids) / len(case.required_ids)


def has_leakage(case: EvaluationCase, result: RetrievalResult, k: int) -> bool:
    return bool(set(result.ranked_ids[:k]) & case.forbidden_ids)


def evaluate(
    cases: Iterable[EvaluationCase],
    results: Iterable[RetrievalResult],
    k: int = 5,
) -> dict[str, dict[str, float]]:
    by_case = {result.case_id: result for result in results}
    totals = defaultdict(lambda: {"cases": 0.0, "recall": 0.0, "leaks": 0.0})

    for case in cases:
        result = by_case[case.case_id]
        bucket = totals[case.slice_name]
        bucket["cases"] += 1
        bucket["recall"] += recall_at_k(case, result, k)
        bucket["leaks"] += float(has_leakage(case, result, k))

    return {
        name: {
            "mean_recall_at_k": values["recall"] / values["cases"],
            "leakage_rate": values["leaks"] / values["cases"],
        }
        for name, values in totals.items()
    }


utc = timezone.utc
cases = [
    EvaluationCase(
        case_id="correction-001",
        query="What release date was confirmed for the Atlas SDK?",
        observed_at=datetime(2026, 8, 14, 12, 0, tzinfo=utc),
        slice_name="corrections",
        required_ids=frozenset({"atlas-correction-v2"}),
        forbidden_ids=frozenset({"atlas-original-v1"}),
        answerable=True,
    ),
    EvaluationCase(
        case_id="retention-031",
        query="Which change preceded the current Atlas SDK policy?",
        observed_at=datetime(2026, 8, 31, 12, 0, tzinfo=utc),
        slice_name="older_followups",
        required_ids=frozenset({"atlas-policy-previous"}),
        forbidden_ids=frozenset(),
        answerable=True,
    ),
]

candidate_results = [
    RetrievalResult("correction-001", ("atlas-correction-v2", "atlas-analysis")),
    RetrievalResult("retention-031", ("atlas-policy-previous", "atlas-current")),
]

print(evaluate(cases, candidate_results, k=5))

The identifiers and dates are synthetic fixture data. The evaluator itself is deterministic, and it can fail a build if a candidate misses a team-defined gate. Keep the gate in a versioned configuration next to the cases; do not bury it in a dashboard filter. Also store the corpus snapshot identifier, chunking configuration, and retrieval configuration with each run, because a score without those inputs cannot be reproduced.

There is a subtle labeling trap here. If assessors see the candidate ranking before marking required evidence, they may bless plausible retrieved text rather than the evidence the question demands. Labels should be created against the source snapshot first, then adjudicated when assessors disagree. I'm not sure a single agreement threshold fits every newsroom or internal developer-tools team; a pilot with double-labeled cases would resolve that local question better than a borrowed percentage.

Run the same cases against multiple retention snapshots: for example, 14, 30, 60, and 90 days of searchable material, with the archive held constant. Record per-slice quality and retained bytes. Now the architecture decision has a frontier: discard any candidate that misses a required slice, then compare resource use among the survivors. Simple. If no candidate passes, improve ingestion, canonicalization, chunking, or ranking before purchasing more retained capacity by reflex.

Choose the smallest passing corpus, then name the loss

The decision rule is intentionally strict: select the least costly candidate that clears every critical slice, not the candidate with the prettiest average. A team may define correction leakage as a hard zero while allowing a small, measured drop in recall for old, low-priority announcements. Those are policy choices, so write them beside the evaluation results and have the knowledge-base owner approve them.

Failure modes deserve names because they point to different fixes. temporal_leakage means the system retrieved information unavailable at the observation time. revision_collapse means deduplication erased a meaningful correction. retention_miss means relevant evidence aged out of the searchable tier. identity_collision means the ranking joined unrelated entities. unsupported_answer means generation proceeded without required evidence. One blended “accuracy” field would erase all five diagnoses.

The catch is that the smallest hot corpus is not suitable when users require low-latency, full-history discovery and cannot tolerate an archive restore. In that case, retain the full searchable history and accept its measured index cost. A short hot window plus an object archive is better suited to workloads where historical questions are rare and slower replay is acceptable. It also carries a real operational burden: restoration must preserve source identity, version order, and observation timestamps, and the team must test how long a rebuild takes using its own corpus.

What do we deliberately stop keeping? Redundant searchable representations of byte-equivalent syndication copies, expired intermediate chunks, and old index segments that fail to improve the age-bucket tests. Keep the authoritative raw objects, provenance, hashes, correction links, and evaluation snapshots according to the organization's retention obligations. The price of the smaller index appears during an investigation: an old question may require a slower archive lookup or a rebuild before it can be answered. That's not free, and pretending otherwise would turn a storage policy into marketing.

Deploy the policy gradually. Shadow the candidate against the current architecture, compare slice metrics on the same queries, and alert on corpus age, duplicate rate, chunks per accepted document, index bytes, archive restore duration, and unsupported-answer rate. Rollback should mean selecting the previous immutable snapshot, not reconstructing an undocumented state from whatever happens to remain in the ingestion queue.

The retrieval architecture follows from this evidence. Keep a durable source layer, derive a reproducible searchable layer, bind every evaluation case to time and provenance, and promote only retention policies that pass the failure-mode slices. The final choice may still vary with query mix and obligations — your mileage may vary — but the disagreement will be about visible costs and visible losses rather than a vendor feature matrix.

References

• Lewis, Patrick et al. “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks.” https://arxiv.org/abs/2005.11401

Further reading

The paper above is the starting point for the retrieved non-parametric memory model. Read it alongside the evaluation artifacts from your own service; the local artifacts determine retention, correction, and temporal-leakage requirements that a general research paper cannot supply.