DEV Community

Cover image for Filter ESPN Data Notices in Python Before Embedding Tennis Matches
vexaloom-labs
vexaloom-labs

Posted on Fully Autonomous

Filter ESPN Data Notices in Python Before Embedding Tennis Matches

Your row limit counts records; your retrieval task counts matches.

Set maxItems low and it's tempting to embed each returned row. For tennis, that row can carry a tournament's nested matches. The context window gets the tournament, even when your question concerns a match.

Lab note 141: can we make a match independently retrievable without copying its siblings into the embedding? That's a schema-level bet. Better retrieval scores would still need evaluation against your questions.

Where the record cap stops

maxItems caps data records, not nested tennis matches or tokens.

Its default is 50, and its allowed range is 1 through 5000. Those bounds control acquisition. They don't establish an embedding budget.

The Actor's tennis scoreboard returns event records with a flat matches[] list alongside groupings[]. The event row is a useful transport container. Making it your retrieval unit is a separate decision.

This Example acquisition selects tennis explicitly and streams the resulting dataset:

import hashlib
import json
import os

import tiktoken
from apify_client import ApifyClient

run_input = {
    "mode": "scoreboard",
    "league": "atp",
    "datePreset": "last7Days",
    "maxItems": 1,
}

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor(
    "vexaloom-labs/espn-scores-standings-teams-and-news-by-league"
).call(run_input=run_input)

if run["status"] != "SUCCEEDED":
    raise RuntimeError(f"Actor ended with status {run['status']}")

rows = client.dataset(run["defaultDatasetId"]).iterate_items()
Enter fullscreen mode Exit fullscreen mode

A successful run still needs content inspection: an unsupported or empty request can produce a notice row. That belongs in your ingestion report, outside the sports corpus.

What does a match need to carry?

Our target task is narrow: retrieve the relevant tennis match, then give an assistant enough context to identify it and describe its available state.

That suggests a projection rather than an indiscriminate tournament dump.

Field or structure Destination Reason
Input mode, league, maxItems Ingestion configuration; copy the selected league into parent metadata Preserve acquisition intent without treating controls as match content
recordType Row routing Separate events from notices and unexpected record types
Event eventId, name, date Parent manifest and every match chunk A retrieved match must remain identifiable away from its original row
Match matchId Chunk identity and match content Identify the child within its event
Match round, court Match content Distinguish encounters within the tournament
Match notes, status Match content, intact Preserve available qualifications and state
Match competitors[] Match content, intact Keep the participants together
matches[] Flatten into child chunks Align the retrieval unit with the question's subject
groupings[] Exclude from embedding text Avoid indexing parallel representations of match material

The catch is inside competitors and status: knowing those containers exist doesn't justify inventing their internal paths.

We preserve them intact. We'd rather carry an opaque object than confidently extract a participant name from a key we guessed.

The event's date also stays under the event namespace. Renaming it to “match start time” would add meaning the field mapping hasn't established.

Keep a projection, not both trees

The intended split looks like this:

event row
  ├─ event identity ───────────────────> parent manifest
  ├─ matches[] ──> match + event context ──> embedding candidate
  └─ groupings[] ──────────────────────> excluded from embedding text
Enter fullscreen mode Exit fullscreen mode

Flattening both tennis representations risks giving repeated material extra retrieval weight. More serialized text doesn't necessarily mean more evidence. Sometimes it means the same evidence brought a folding chair.

We'd keep the original event payload in a separate raw-data archive if reprocessing matters. The embedding projection has a narrower job.

Some repetition is useful: the tournament identity travels with each match. Repeating a small parent envelope buys independence; repeating sibling matches changes what the chunk is about.

The trade-off appears in broader questions. A tournament-wide recap may need several retrieved matches or an event-level aggregation. Match chunks are a deliberate fit for match retrieval, not a universal replacement for every sports query.

Count the text you actually embed

The serializer below defines the flat-list field explicitly, preserves the selected fields, and measures the exact string intended for embedding.

Its token budget and encoding are explicit application choices. Match the encoding to your embedding system before treating the count as its operational limit.

MATCH_LIST_FIELD = "matches"
EVENT_FIELDS = ("eventId", "name", "date")
MATCH_FIELDS = (
    "matchId", "round", "court", "notes", "status", "competitors"
)

encoding = tiktoken.get_encoding("cl100k_base")
token_budget = 1200

def compact(value):
    return json.dumps(
        value, ensure_ascii=False, sort_keys=True, separators=(",", ":")
    )

def project(record, fields):
    return {field: record[field] for field in fields if field in record}

def stable_key(*parts):
    return hashlib.sha256(compact(parts).encode("utf-8")).hexdigest()

def serialize_match(parent, match):
    text = compact({
        "event": parent,
        "match": project(match, MATCH_FIELDS),
    })
    return {
        "chunk_id": stable_key(
            parent["league"], parent["eventId"], match["matchId"]
        ),
        "metadata": {**parent, "matchId": match["matchId"]},
        "text": text,
        "token_count": len(
            encoding.encode(text, disallowed_special=())
        ),
    }
Enter fullscreen mode Exit fullscreen mode

The chunk key combines the selected league, event identity, and match identity. It makes no assumption that matchId is globally unique.

That distinction matters during refreshes. A match's status can change without becoming a different match. Use the stable key for corpus updates; using a content hash as the only identity would turn each changed snapshot into another retrievable version.

The count covers text. Metadata, the JSONL envelope, and instructions added to a later generation prompt need their own accounting.

Write match chunks and an event manifest

This cell writes accepted chunks, parent records, and a review queue. Oversized candidates retain their complete content for a later structured split.

parents = {}
chunks = {}

with open("review.jsonl", "w", encoding="utf-8") as review_file:
    def route(reason, record):
        review_file.write(
            compact({"reason": reason, "record": record}) + "\n"
        )

    for row in rows:
        record_type = row.get("recordType")
        if record_type == "notice":
            route("notice", row)
            continue
        if record_type != "event":
            route("unexpected_record_type", row)
            continue
        if row.get("eventId") in (None, ""):
            route("missing_event_id", row)
            continue

        parent = {
            "league": run_input["league"],
            **project(row, EVENT_FIELDS),
        }
        event_key = stable_key(parent["league"], parent["eventId"])
        parents[event_key] = {"event_key": event_key, **parent}

        matches = row.get(MATCH_LIST_FIELD)
        if not isinstance(matches, list) or not matches:
            route("missing_matches", row)
            continue

        for match in matches:
            if (
                not isinstance(match, dict)
                or match.get("matchId") in (None, "")
            ):
                route("missing_match_id", {
                    "event": parent, "match": match
                })
                continue

            chunk = serialize_match(parent, match)
            chunks[chunk["chunk_id"]] = chunk

    with open("matches.jsonl", "w", encoding="utf-8") as match_file:
        for chunk in chunks.values():
            if chunk["token_count"] > token_budget:
                route("over_token_budget", chunk)
                continue
            match_file.write(compact(chunk) + "\n")

with open("events.jsonl", "w", encoding="utf-8") as event_file:
    for parent_record in parents.values():
        event_file.write(compact(parent_record) + "\n")
Enter fullscreen mode Exit fullscreen mode

The dictionaries retain the last encountered version of each key within this batch. That's an ordering policy, not a freshness guarantee. Across ingestions, store an observation timestamp and define which snapshot may replace an existing record.

This version also buffers projected chunks in memory. For a larger corpus, move that keyed staging step into persistent storage rather than accumulating every candidate in a process.

At embedding time, submit each accepted record's text and retain its chunk_id and metadata. Sending the entire JSONL envelope would repeat context the serializer already included.

What refuses to fit

Cutting serialized JSON at a token boundary doesn't work as a semantic policy. It can remove a competitor, detach a status qualification, or leave invalid JSON. A smaller string isn't automatically a smaller fact.

The review queue gives oversized matches an explicit destination. A follow-up splitter can repeat event identity, complete competitors, and status while separating independently meaningful notes.

If the mandatory context alone exceeds your budget, keep that match out of the embedding batch until you change the budget or design a smaller, verified projection.

Requesting gameDetail doesn't supply a richer tennis fallback either. These individual-sport scoreboards don't expose the same team-style summary; that request returns a notice.

Missing fields also remain missing. The Actor omits empty values, so the serializer can't turn absent status or participant detail into evidence. Your answer layer needs permission to say the record doesn't establish something.

The event transports; the match retrieves

For this task, we'd embed the match projection and keep the event as its parent.

The flat list gives us a suitable boundary. It doesn't establish that every match will fit, that every question is match-shaped, or that retrieval quality improves without evaluation.

ESPN Scores, Standings, Teams & News by League is the worked example. The reusable decision is to choose the embedding unit from the question's subject.

For a retrieval evaluation, check whether the relevant match appears, whether nearby matches crowd it out, and whether the assistant can distinguish missing evidence from a negative answer. A compact chunk can still be the wrong chunk.

Reproduce the boundary

  1. Install apify-client and tiktoken, and set APIFY_TOKEN in your environment.
  2. Run the Python cells in order with the input below.
  3. Inspect events.jsonl, matches.jsonl, and review.jsonl separately. Empty requests belong in the review path.
  4. Feed accepted text values into your embedding pipeline and retain their stable identities for updates.

Use this exact Example input:

{
  "mode": "scoreboard",
  "league": "atp",
  "datePreset": "last7Days",
  "maxItems": 1
}
Enter fullscreen mode Exit fullscreen mode

Drafted with AI assistance and reviewed for accuracy.

Top comments (0)