A pagination scheme is only correct if the source can change while the importer is reading it. Short answer: paginate financial research documents with a stable compound key, freeze each ingestion run at a source watermark, and publish the new vector-index generation only after every page is accounted for. This makes freshness measurable without letting an update create a duplicate or make a document vanish between pages.
That choice matters in a financial research dashboard because the source set is moving: revised notes, new filings, and corrected support documents may arrive during ingestion. Retrieval-augmented generation combines retrieved external material with generation; the ingestion path therefore deserves the same operational attention as the query path. A good answer generated from yesterday's complete index can be safer than an answer assembled from a half-refreshed index whose coverage nobody can explain.
The mental model is short. Before: fetch page 1, fetch page 2, and hope offsets still mean the same thing. After: declare a batch boundary, walk an ordered keyspace, record evidence for each page, then swap one complete generation into service.
How should a financial research dashboard paginate retrieval without stale context?
Use keyset pagination over a unique, immutable ordering tuple such as (sourceUpdatedAt, documentId). At the beginning of a run, capture a watermark from the source. Every page must satisfy two bounds: its key is greater than the last key already processed, and its update time is less than or equal to that watermark. The first bound moves the reader forward. The second prevents documents arriving mid-run from changing the membership of the batch.
The unique tie-breaker is not decoration. Two documents can share the same update timestamp, especially when a back-office export assigns time in batches. Ordering by time alone leaves their relative order undefined. Pairing time with an immutable document ID gives every record one exact position, while the cursor carries both values into the next request.
Keep the cursor opaque at the transport boundary even if its decoded form is simple. Clients should return it, not edit it. Internally, treat a page as an envelope containing items, nextCursor, snapshotWatermark, and a stable batchId. An empty nextCursor means the bounded scan is complete; it should not mean "try another offset and see."
This pattern also answers a subtle freshness question. A document updated one millisecond after the watermark belongs to the next batch. It isn't lost. It is deferred by an explicit rule that an operator can inspect, alert on, and explain.
Turn the page loop into an ingestion contract
Here is a compact TypeScript example. It uses an in-memory source so the file runs without an SDK, but the contract maps directly to a database query or an internal service: filter to the watermark, compare the compound key, sort in the same direction every time, and request one extra item to determine whether another page exists.
type SourceDocument = {
id: string;
updatedAt: string;
title: string;
body: string;
};
type Cursor = Pick<SourceDocument, "updatedAt" | "id">;
type Page = {
items: SourceDocument[];
nextCursor?: Cursor;
snapshotWatermark: string;
};
const documents: SourceDocument[] = [
{
id: "doc-104",
updatedAt: "2026-08-31T09:30:00.000Z",
title: "Credit review FAQ",
body: "Analyst support guidance for the quarterly credit review."
},
{
id: "doc-105",
updatedAt: "2026-08-31T09:30:00.000Z",
title: "Issuer note correction",
body: "Corrected research note supplied to the support desk."
},
{
id: "doc-109",
updatedAt: "2026-08-31T12:00:00.000Z",
title: "Sector brief",
body: "Research brief for dashboard retrieval."
}
];
function compareKey(left: Cursor, right: Cursor): number {
return left.updatedAt.localeCompare(right.updatedAt) || left.id.localeCompare(right.id);
}
function readPage(
watermark: string,
after: Cursor | undefined,
pageSize: number
): Page {
const eligible = documents
.filter((document) => document.updatedAt <= watermark)
.filter((document) => !after || compareKey(document, after) > 0)
.sort(compareKey);
const window = eligible.slice(0, pageSize + 1);
const items = window.slice(0, pageSize);
const hasNextPage = window.length > pageSize;
const last = items.at(-1);
return {
items,
nextCursor: hasNextPage && last
? { updatedAt: last.updatedAt, id: last.id }
: undefined,
snapshotWatermark: watermark
};
}
async function ingestBatch(watermark: string): Promise<void> {
let cursor: Cursor | undefined;
const seen = new Set<string>();
do {
const page = readPage(watermark, cursor, 2);
for (const document of page.items) {
if (seen.has(document.id)) throw new Error(`duplicate document: ${document.id}`);
seen.add(document.id);
// Replace this line with deterministic chunking and an idempotent index upsert.
console.log({ documentId: document.id, batchWatermark: page.snapshotWatermark });
}
cursor = page.nextCursor;
} while (cursor);
}
void ingestBatch("2026-08-31T12:00:00.000Z");
The important part isn't the array. It is the invariant expressed by the two filters. In a persistent store, those predicates and the sort must use the same fields and directions. The corresponding index should begin with the fields used by that ordered scan; validate the actual query plan under representative volume rather than assuming an index name proves anything.
Now add chunking. Derive each chunk ID from the source document ID, a content revision, and the chunk's ordinal position. Reprocessing the same page then produces the same identities, so an upsert repairs an interrupted run instead of multiplying chunks. Delete or retire chunks from the previous revision only when the complete replacement revision is ready. Fast retries are useful. Invisible duplication is not.
Measure freshness as a pipeline, not a timestamp
One lastIndexedAt gauge hides too much. The useful diagram-in-words is: source watermark -> page fetch -> document validation -> deterministic chunking -> embedding -> index upsert -> generation publish -> retrieval check. Put a duration and outcome at each arrow. Attach batchId, the hashed cursor position, page size, item count, and watermark to logs or traces; keep document bodies and full query text out of telemetry because they may contain sensitive research or support content.
Track three distinct signals because one freshness number cannot answer three operational questions:
| Signal | What it compares | Question it answers |
|---|---|---|
| Source lag | Newest eligible source update and the active generation's watermark | Are we behind? |
| Batch age | Batch start and the current time | Are we stuck? |
| Coverage | Accepted source documents and documents represented in the candidate generation | Did we omit anything? |
Count outcomes too: pages read, documents accepted, unchanged documents skipped, chunks written, validation failures, retries, and duplicate-key rejections. A page can return quickly while every item fails validation. Latency alone would paint that run green.
Alert on sustained conditions rather than a single slow page. Set thresholds from the dashboard's publication promise and observed source cadence; no universal number is defensible here. A pre-market research workflow and an archive of support guidance carry different freshness costs. I'm not sure which threshold fits a given desk until its owners define how late content may be before the dashboard must disclose it or stop serving it.
Then test the alarms.
Feed a fixture in which doc-104 and doc-105 both have the timestamp 2026-08-31T09:30:00.000Z, then read a one-item first page. Between page reads, add doc-106 with a time after the batch watermark and revise the body of doc-105 without changing the frozen test fixture seen by this batch. Resume from the full (updatedAt, id) cursor, retry that same page once, and stop the worker immediately before publication. Now inspect identities and state, not just returned arrays: doc-104 and doc-105 must each occur once in the candidate generation, every repeated upsert must target the same chunk IDs, doc-106 must wait for the next watermark, and the active generation must still point at the previous complete batch. Finally, resume the interrupted batch and verify that publication advances exactly once. This single exercise probes equal timestamps, a concurrent arrival, retry idempotency, interruption recovery, and atomic visibility. A happy-path screenshot proves none of those properties.
Make this test deterministic.
The catch: snapshots, deletions, and very large batches
A watermark is not a database snapshot. If an existing record can be deleted or have its ordering fields rewritten during the scan, keyset pagination alone cannot reconstruct the earlier set. Choose one of three explicit policies: read from a true snapshot, consume an append-only change log, or reconcile the completed candidate generation against a source manifest before publication. The right option depends on retention rules and how much history the source exposes.
This design is not suitable when the source cannot provide a stable unique key or any trustworthy change marker. In that case, a full immutable export with a checksum is a better boundary. Stick with a transactional snapshot when strict point-in-time membership is required and the datastore can hold that snapshot for the whole read. Use a change stream when near-real-time ingestion matters more than periodic, bounded batches. Each option moves complexity; none removes it.
Very large batches add another trade-off. A frozen watermark makes the run explainable, but a six-hour run can still miss a freshness objective even while it is perfectly correct. Reduce page work, parallelize only across disjoint key ranges, or publish smaller complete partitions whose boundaries are visible to retrieval. Don't let workers share a mutable cursor. That creates speed in a load test and ambiguity during recovery.
Deletion needs a first-class event or a manifest diff. Otherwise a removed research note can remain retrievable forever, even though every pagination metric looks healthy. Mark tombstones with the same batch identity, apply them to the candidate generation, and include their count in the publish check.
Two objections worth resolving before deployment
"Why not use offsets?" Offsets are easy to inspect and fine for a small, immutable result set. They become a weak ingestion checkpoint when earlier rows can be inserted or removed, because the meaning of offset 200 can change between requests. A compound cursor anchors progress to the last observed record. The catch is that ad hoc jumping to page 37 becomes awkward, which is acceptable for a sequential ingestion worker but may be unacceptable for a human-facing table. Use separate pagination contracts for those two jobs.
"Why not update the live index after every page?" That minimizes visibility delay, but readers can observe a mixed generation: some revised documents, some old chunks, and possibly two revisions of one source. Publishing a complete generation gives retrieval one coherent boundary. If storage or indexing cost makes whole-generation publication impractical, partition by a domain key and expose each partition's watermark so the query layer can reject or label inconsistent combinations. There is no free choice here — freshness, coherence, storage, and recovery time pull in different directions.
The deployment gate should be boring: the bounded scan ended, accepted plus rejected equals observed, every accepted revision has its expected chunk identities, tombstones were applied, retrieval smoke checks passed, and the candidate watermark moved forward. Only then switch the active generation. Pagination is finished when the evidence closes, not when the final request returns an empty cursor.
References
- Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks: https://arxiv.org/abs/2005.11401
Top comments (0)