Exporting a paginated task feed without losing provenance

A JSON endpoint is easy to download. A complete, attributable snapshot takes a little more care.

I maintain Instinct Tasks, an independent, unofficial guide. Its public /api/tasks endpoint returns an envelope containing protocolVersion, snapshot, items, nextCursor and checkpoint. In the October 10 export, the first response contained 20 items. Treating that response as the whole library would have missed most of the 128 records.

The export therefore follows nextCursor until it is empty. Every response must have the same snapshot identifier. If that identifier changes, the export should stop and be investigated or restarted rather than silently mixing versions.

There are three useful integrity checks after pagination:

• Task IDs are unique across the collected responses.
• Each task URL belongs to the expected guide domain.
• Each record retains its source URLs and evidence type.

The first check catches duplicate pages. The second catches a wrong-domain export. The third preserves the distinction between a community report and an example published by the product maker. None of these checks proves that the underlying task works; they verify the structure and provenance of the export.

Here is a small local inspection of the completed snapshot:

import json
from collections import Counter
from pathlib import Path

snapshot = json.loads(Path("tasks.json").read_text())
tasks = snapshot["items"]
assert len(tasks) == len({task["taskId"] for task in tasks})
print(Counter(task["evidenceType"] for task in tasks))
for task in tasks:
    print(task["title"], task["url"], task["sources"])

For this export, the evidence labels were 124 user reports and four official examples. They are counts of published records, not trial results or adoption statistics.

The snapshot also records its retrieval time, original endpoint and snapshot ID. A checksum in the README identifies the exact file. That makes it possible to discuss a fixed version even as the website changes.

The resulting reference repository is https://github.com/1021ju/instinct-tasks. It contains published metadata, not private tasks or execution logs. Source links are retained, and no blanket open-source license is asserted for third-party materials. The guide is not affiliated with Instinct and has not independently reproduced the source examples.