Your publisher allowlist can admit articles from publishers you never approved.
Picture your retrieval evaluation: the answer looks sensible, the citation opens correctly, and the publisher isn't on your approved list. Your ingestion job accepted the cluster. Retrieval inherited the guests.
Lab note 098: does approving a story cluster also approve the articles inside it? The catch comes before embeddings: includeDomains selects clusters, while your source policy needs to select individual evidence records.
What does the publisher allowlist actually approve?
The news-cluster Actor keeps a cluster when at least one linked article matches includeDomains. With includeArticles enabled, it emits linked article records from the admitted cluster, subject to maxArticlesPerStory.
A publisher allowlist applied to story clusters does not restrict every emitted article to an approved publisher.
That distinction is easy to miss when the configuration looks like an ordinary row filter. The admission decision belongs to the parent; the records you embed belong to its children.
This Example fixture makes that boundary visible without pretending it came from a collection run.
fixture = [
{
"recordType": "story",
"storyId": "example-cluster",
"headline": "Example event coverage",
},
{
"recordType": "article",
"storyId": "example-cluster",
"articleUrl": "https://approved.example/news",
"headline": "Example approved headline",
"summary": "Example approved snippet.",
"domain": "ru",
},
{
"recordType": "article",
"storyId": "example-cluster",
"articleUrl": "https://outside.example/report",
"headline": "Example unapproved headline",
"summary": "Example unapproved snippet.",
"domain": "ru",
},
]
The collection decision and the evidence decision need separate gates:
cluster with an approved article
|
v
cluster admitted
|
+--> approved article --> URL policy --> embed
|
+--> other article ----> URL policy --> reject
If you embed first and filter citations later, the rejected source may already have influenced retrieval or an intermediate summary. Removing its citation doesn't remove its contribution.
Story rows can smuggle aggregate text into evidence
The dataset contains story records alongside article records. Story headlines and summaries describe the cluster; they aren't automatically attributable to an approved publisher.
Keeping a story because it has an approved child doesn't establish publisher provenance for that aggregate text. Embedding it as publisher evidence gives your model a convenient claim with an inconvenient attribution problem.
Start by admitting only article records to the evidence path.
def article_rows(rows):
for row in rows:
if row.get("recordType") == "article":
yield row
Keep story records in a separate navigation or analysis store if they're useful. Retain each accepted article's storyId so you can trace the relationship without embedding the parent's summary.
This also protects against turning collection metadata into searchable prose. A record's presence in the dataset isn't a reason to serialize every field into an embedding.
The publisher lives in articleUrl
The output's domain field is the constant ru. It isn't the publisher hostname.
Filtering on that field can't enforce a publisher policy. sourceName is useful display metadata, but a name match isn't a URL identity check either.
Parse articleUrl and compare its hostname against an explicit policy. Substring checks can accept a hostname containing your approved name, or a URL that merely mentions it in the path.
This Example policy approves an exact hostname and leaves subdomain permission disabled.
from urllib.parse import urlsplit
EXACT_HOSTS = frozenset({"approved.example"})
SUBDOMAIN_ROOTS = frozenset()
def eligible_url(value):
if not isinstance(value, str) or not value:
return None
if "\\" in value or any(
char.isspace() or ord(char) < 32 for char in value
):
return None
try:
parsed = urlsplit(value)
host = (parsed.hostname or "").encode("idna").decode("ascii")
host = host.lower().removesuffix(".")
parsed.port # Validate port syntax.
except (ValueError, UnicodeError):
return None
if parsed.scheme.lower() not in {"http", "https"}:
return None
if parsed.username is not None or parsed.password is not None:
return None
allowed = host in EXACT_HOSTS or any(
host == root or host.endswith("." + root)
for root in SUBDOMAIN_ROOTS
)
return value if allowed else None
Use lowercase ASCII hostnames in the policy. Add a root to SUBDOMAIN_ROOTS only when your approval actually covers its subdomains.
This checks the supplied URL. It doesn't certify ownership, follow redirects, or approve a destination reached later. If you subsequently fetch article bodies, apply the source policy again to the final response URL before admitting that text.
There's another tempting fix that doesn't work: filling excludeDomains with unwanted outlets. That setting drops a cluster when any linked article matches an excluded domain.
For an approved-evidence corpus, that can discard useful approved coverage along with the unwanted sibling. Use cluster exclusions when you intend to reject the event's entire coverage bundle. Use the local URL gate when you intend to reject individual articles.
A snippet can't carry a full article's weight
The Actor returns article metadata and publisher URLs, not full article text. An article summary, when present, is a snippet.
Passing those fields through a generic article chunker can make a thin record look like a complete source. Overlapping chunks may repeat the same short passage, spending token budget without adding evidence. A confident answer can then outrun the material retrieved.
Project only the available headline and snippet into the text path; reject records with neither.
def snippet_fields(row):
fields = {}
for name in ("headline", "summary"):
value = row.get(name)
if isinstance(value, str) and value.strip():
fields[name] = value.strip()
return fields
Treat that projection as snippet evidence. Don't ask it to support details that require the article body.
For this corpus, we'd keep the short projection intact before considering overlapping chunks. If you need full-text answers, body acquisition is a separate ingestion stage with its own access and source checks.
Put eligibility before deduplication and embeddings
The combined filter should produce evidence documents, preserve parent relationships, and avoid collapsing an entire event into whichever article happens to arrive first.
The Actor deduplicates by story ID and article ID. That doesn't establish text uniqueness. Conversely, deduplicating your corpus by storyId can discard distinct publisher evidence about the same event.
Here, the local deduplication key is the accepted URL plus its projected text. Repeated copies share parent IDs; changed snippets remain separate. This is deliberately conservative: different URLs aren't assumed to be equivalent.
import json
def collect_documents(rows):
accepted = {}
for row in article_rows(rows):
url = eligible_url(row.get("articleUrl"))
story_id = row.get("storyId")
fields = snippet_fields(row)
if url is None or not fields:
continue
if not isinstance(story_id, str) or not story_id.strip():
continue
text = "\n".join(fields.values())
key = (url, text)
if key not in accepted:
document = {
"recordType": "article",
"articleUrl": url,
**fields,
}
source_name = row.get("sourceName")
if isinstance(source_name, str) and source_name.strip():
document["sourceName"] = source_name.strip()
accepted[key] = (document, set())
accepted[key][1].add(story_id)
documents = []
for document, parents in accepted.values():
documents.append({**document, "storyId": sorted(parents)})
return documents
fixture_documents = collect_documents(fixture)
assert any(
eligible_url(row.get("articleUrl"))
for row in article_rows(fixture)
)
assert {doc["articleUrl"] for doc in fixture_documents} == {
"https://approved.example/news"
}
fixture_jsonl = "".join(
json.dumps(doc, ensure_ascii=False) + "\n"
for doc in fixture_documents
)
In this derived document format, storyId is a list of parent IDs. The assertions check the intended boundary: the synthetic cluster qualifies for admission, while its unapproved article doesn't reach the evidence documents.
The JSONL contains only accepted projections. Send those through your tokenizer and embedding stage, with URLs and parent IDs attached as metadata.
Approval belongs to the evidence retrieval can return
The Yandex News Story Clusters Across Russia & the CIS Actor is the worked example; the ingestion mistake applies anywhere a parent-level filter expands into child records.
The verdict is plain: cluster admission is a discovery decision. Publisher approval must happen again on each article before embedding.
Keep rejection counts by reason in a production pipeline. That lets you distinguish an empty approved corpus from a collection failure without quietly loosening the policy. Failing closed is useful; silently producing nothing is a different annoyance.
Reproduce the boundary before connecting retrieval
Run the Python cells in order to check the synthetic admission case. Then replace the reserved Example hostname in both the policy and the input below with an approved publisher hostname.
The following Example collection uses the same gate on dataset records and builds JSONL in memory.
import os
from apify_client import ApifyClient
run_input = {
"mode": "searchNews",
"searchQueries": ["энергетика"],
"includeDomains": ["approved.example"],
"excludeDomains": [],
"includeArticles": True,
"maxStories": 20,
"maxArticlesPerStory": 100,
}
client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
"vexaloom-labs/yandex-news-story-clusters-across-russia"
).call(run_input=run_input)
if run["status"] != "SUCCEEDED":
raise RuntimeError(f"Collection ended with status {run['status']}")
documents = collect_documents(
client.dataset(run["defaultDatasetId"]).iterate_items()
)
jsonl = "".join(
json.dumps(doc, ensure_ascii=False) + "\n"
for doc in documents
)
Before indexing, check that every document URL passes your policy and that rejected URLs never enter the embedding queue. An empty accepted set is valid; it isn't permission to fall back to unfiltered rows.
Use this exact Example input for the collection configuration, replacing the reserved hostname before a live collection:
{
"mode": "searchNews",
"searchQueries": ["энергетика"],
"includeDomains": ["approved.example"],
"excludeDomains": [],
"includeArticles": true,
"maxStories": 20,
"maxArticlesPerStory": 100
}
Drafted with AI assistance and reviewed for accuracy.
Top comments (0)