Your Section Anchors Don't Locate Your RAG Chunks

Your retriever finds a useful passage. Then canonicalUrl + "#" + anchor sends the reader to a heading that the passage may never have belonged to.

The link looks precise. That's the annoying part.

Lab note 085: can we build section-aware retrieval from fullText and sections alone? Those fields provide article text and a heading outline, but neither connects headings to ranges in the body.

Why matching a heading can lie

The tempting repair is to search the body for each heading title, then slice between matches.

That quietly assumes a heading phrase can't appear earlier in ordinary prose. Your ingestion job has just acquired a literary opinion.

Example synthetic text makes the ambiguity visible without fetching an article:

heading_phrase = "History"
synthetic_text = (
    "Example: History is mentioned in this opening paragraph.\n\n"
    "History\n"
    "The section account starts here."
)

naive_start = synthetic_text.find(heading_phrase)
known_start = synthetic_text.index(
    "\n\n" + heading_phrase + "\n"
) + len("\n\n")

assert naive_start != known_start

The assertion checks a fixture whose boundary we deliberately placed. It says nothing about the formatting of a returned article.

That distinction matters: recognizing a heading-shaped string is a heuristic. Establishing which passage belongs beneath it requires more evidence.

What should a retrieved passage promise?

You're building a corpus where a chunk's heading and citation should describe the text it actually contains.

Our bet: the model needs passage text and article identity to answer with an article citation. An outline can help explain the article's organization, but it can't supply passage ownership.

The lab's article lookup Actor makes the distinction concrete. Its body text and outline are separate outputs.

This is the relationship your ingestion code can safely preserve:

article identity ──┬── body text ── token windows ── article citations
                   └── heading outline ── navigation context

heading outline ── ? ── body ranges

The missing connection is the experiment.

A heading label attached to a chunk can influence retrieval before an LLM ever sees it. Embed an unrelated heading alongside a passage and you've changed the search representation, not merely decorated the result.

The fields we actually need

Here's the field map for that task. “Use” describes our serializer's decision, rather than an extra promise from the Actor.

Field	Use in our pipeline	Boundary it cannot establish
includeFullContent, includeSections	Enable the separate outputs; both default to false	Requesting them together does not create a mapping
fullText	Passage source	Where a particular section begins or ends
sections[].title, sections[].level	Build an outline hierarchy	Which body characters belong beneath that hierarchy
sections[].anchor	Retain navigation metadata	Which chunk can cite that fragment
title	Label the article	A passage's section title
canonicalUrl	Cite the source article	A historical snapshot
revisionId	Retain revision identity when present	A section boundary or a revision-specific citation URL

The distinction between navigation and evidence is the useful part.

A fragment can identify a destination on a page while offering no proof that a selected passage belongs there. A working link can't validate the metadata attached to your embedding.

Keep the outline out of passage evidence

We'd keep the outline alongside the article and index the body chunks independently.

When a retrieval result needs broader context, the outline can travel once with its source article. It should be identified as an outline, rather than presented as headings belonging to every retrieved passage.

For this task, we'd drop displayTitle, images, coordinates, pageviews and search provenance from the answer prompt. Some belong in a catalog or audit trail; they don't establish passage ownership.

We'd also avoid appending summary to the body before chunking. It's lead-section text, so adding it creates another representation of introductory material and can overweight that material during retrieval.

Deduplication needs a separate decision. Identical text under different source articles may still require distinct attribution. A content hash alone is an awkward place to erase that distinction.

A serializer that refuses to guess

The implementation below keeps article identity, local chunk order and an optional revision tag. Its generated article, outline, chunks, path and chunk_id keys belong to our application.

Example lookup input requests the material needed for the teardown:

import hashlib
import json
import os

import tiktoken
from apify_client import ApifyClient

run_input = {
    "articleTitles": ["Python (programming language)"],
    "language": "en",
    "maxItems": 1,
    "includeFullContent": True,
    "includeSections": True,
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "vexaloom-labs/wikipedia-article-lookup-by-url-or-keyword"
).call(run_input=run_input)

if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Actor status: {run['status']}")

items = client.dataset(run["defaultDatasetId"]).list_items().items
encoding = tiktoken.get_encoding("cl100k_base")

The outline builder uses heading levels to maintain ancestry and retains anchors without attaching them to passages:

def outline_context(article):
    entries = article.get("sections")
    if not isinstance(entries, list):
        return []

    ancestry = []
    outline = []

    for entry in entries:
        if not isinstance(entry, dict):
            continue

        label = entry.get("title")
        try:
            level = int(entry.get("level"))
        except (TypeError, ValueError, OverflowError):
            continue

        if not isinstance(label, str) or not label or level < 1:
            continue

        while ancestry and ancestry[-1][0] >= level:
            ancestry.pop()
        ancestry.append((level, label))

        heading = {
            "title": label,
            "level": level,
            "path": [name for _, name in ancestry],
        }
        anchor = entry.get("anchor")
        if isinstance(anchor, str) and anchor:
            heading["anchor"] = anchor
        outline.append(heading)

    return outline

Ancestry here means relationships among headings. It still says nothing about body ranges.

The chunker works with token windows while backing off any boundary that would split a Unicode character. The serializer never adds a section label or fragment citation:

def text_windows(body, budget=256):
    if budget < 4:
        raise ValueError("Use a larger token window")

    token_ids = encoding.encode(body, disallowed_special=())
    start = 0

    while start < len(token_ids):
        end = min(start + budget, len(token_ids))

        while end > start:
            raw = b"".join(
                encoding.decode_single_token_bytes(token)
                for token in token_ids[start:end]
            )
            try:
                passage = raw.decode("utf-8")
            except UnicodeDecodeError:
                end -= 1
                continue
            break
        else:
            raise ValueError("Cannot form a Unicode-safe window")

        yield passage
        start = end

def serialize_article(article):
    body = article.get("fullText")
    source_url = article.get("canonicalUrl")

    if article.get("isDisambiguation"):
        return None
    if not isinstance(body, str) or not body.strip():
        return None
    if not isinstance(source_url, str) or not source_url:
        return None

    identity = {"canonicalUrl": source_url}
    title = article.get("title")
    if isinstance(title, str) and title:
        identity["title"] = title
    revision = article.get("revisionId")
    if revision is not None:
        identity["revisionId"] = revision

    chunks = []
    for chunk_index, passage in enumerate(text_windows(body)):
        fingerprint = json.dumps(
            [identity, chunk_index, passage],
            ensure_ascii=False,
            sort_keys=True,
        )
        chunks.append({
            **identity,
            "chunk_id": hashlib.sha256(
                fingerprint.encode("utf-8")
            ).hexdigest(),
            "chunk_index": chunk_index,
            "text": passage,
        })

    return {
        "article": identity,
        "outline": outline_context(article),
        "chunks": chunks,
    }

Including source identity in the fingerprint preserves attribution when passages share text. Including passage text makes the identifier sensitive to changed content even when revision metadata is absent.

This final cell prepares corpus envelopes and a prompt serializer for passages your retrieval layer selects:

corpus = []
for item in items:
    envelope = serialize_article(item)
    if envelope is not None:
        corpus.append(envelope)

def serialize_prompt(envelope, retrieved_chunks):
    return json.dumps(
        {
            "instruction": (
                "Answer from the supplied passages. "
                "Cite each passage's canonicalUrl. "
                "The outline is article-level navigation context; "
                "do not infer passage section membership from it."
            ),
            "article": envelope["article"],
            "outline": envelope["outline"],
            "passages": retrieved_chunks,
        },
        ensure_ascii=False,
    )

The token window controls passage size. Your retrieval layer must separately budget selected passages, metadata, instructions and answer space.

Where the remaining shortcuts fail

These windows can split sentences or arguments. That's a retrieval-quality trade-off, and it deserves evaluation against your questions. Sentence-aware packing could improve readability without solving section attribution.

The serializer also excludes records without usable body text or a canonical citation. In production, count those exclusions. Quietly substituting a lead paragraph would change the corpus's coverage while leaving its interface looking reassuringly similar.

Malformed outline entries are skipped independently. A bad heading shouldn't manufacture a boundary or prevent useful article text from being indexed.

Rendered HTML is outside this Actor's output. If precise section ownership is a requirement, introduce a source representation that explicitly associates headings with bodies, then verify its text against the passage source.

Store any verified ranges with their source revision. Text transformations and edits can invalidate offsets even when the heading names remain familiar.

What this schema can support

For the worked example, Wikipedia Article Lookup by URL or Keyword supports article-text retrieval with an accompanying outline.

Our schema verdict is plain: these fields support honest article citations, but they can't justify section citations for arbitrary chunks.

That isn't a retrieval benchmark. It's a limit on what the available relationships let a serializer claim.

Keeping revisionId helps track source changes, but the canonical link still points to the article. Retain the captured text and its attribution when you need reproducible evidence; a revision tag beside a live URL isn't an archived passage.

We'd rather omit a section label than let a confident-looking fragment smuggle a guess into the corpus.

Reproduce the boundary check

• Run the synthetic assertion to check why first-match heading assignment fails.
• Set APIFY_TOKEN, install apify-client and tiktoken, then execute the Python cells in order.
• Inspect the generated envelopes: outline paths stay at article scope, while chunks retain source identity and local order.
• Index the chunks, then pass retrieved chunks and their source envelope to serialize_prompt.

Example input matching run_input above:

{
  "articleTitles": ["Python (programming language)"],
  "language": "en",
  "maxItems": 1,
  "includeFullContent": true,
  "includeSections": true
}

The fragment can wait for evidence.