DEV Community: mian po

Engineering Reliable Pagination for Public TikTok Data

mian po — Wed, 29 Jul 2026 01:16:52 +0000

TikTok's public web endpoints do not always return the page size an application requests. One route may return a small native window, another may underfill a page even though more data exists, and a cursor may represent more than a simple numeric offset.

That becomes a production problem quickly:

count can disagree with the number of returned records.
An application can skip items by calculating its own cursor.
Consecutive upstream windows can contain the same item.
A response can claim that more data exists without giving the client a safe continuation value.
Missing booleans can be silently converted to false, changing their meaning.

While building a public TikTok metadata API, I treated pagination as an application contract rather than a thin pass-through to one upstream request.

Define the response contract first

Every list response follows these invariants:

page.count === page.data.length;
page.data.length <= requestedCount;

The response also contains:

{
  "request_id": "correlation-id",
  "data": [],
  "count": 0,
  "cursor": null,
  "has_more": false
}

The important rules are:

count describes the records actually returned, not the requested page size.
has_more=true means the returned cursor can be used for another request.
The client copies the cursor unchanged.
A final page may contain fewer items than requested.
The service never pads a page with duplicate records.

Numeric and opaque cursors are different

Comments and comment replies expose TikTok's numeric continuation cursor. It may look like an offset, but the safest client still treats it as server-owned state.

Creator videos, music videos, hashtag feeds, search, and Explore/trending results use a signed opaque cursor. The opaque value can preserve:

The upstream continuation state
Records left over after filling the previous client page
IDs already emitted while native windows were merged
The resource or search query to which the cursor belongs

A cursor returned for one creator or keyword cannot be reused for another.

Build one reusable client

This Node.js 18+ example calls the API through RapidAPI and retries temporary failures:

const BASE_URL =
  "https://tiktok-public-metadata-api.p.rapidapi.com";

const headers = {
  "X-RapidAPI-Key": process.env.RAPIDAPI_KEY,
  "X-RapidAPI-Host": "tiktok-public-metadata-api.p.rapidapi.com"
};

const sleep = (milliseconds) =>
  new Promise((resolve) => setTimeout(resolve, milliseconds));

async function getPage(path, parameters, attempt = 0) {
  const url = new URL(path, BASE_URL);

  for (const [name, value] of Object.entries(parameters)) {
    if (value !== undefined && value !== null) {
      url.searchParams.set(name, String(value));
    }
  }

  const response = await fetch(url, { headers });

  if ((response.status === 429 || response.status === 503) && attempt < 4) {
    const retryAfter = Number(response.headers.get("retry-after"));
    const delay = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 500 * 2 ** attempt + Math.random() * 250;

    await sleep(delay);
    return getPage(path, parameters, attempt + 1);
  }

  const body = await response.json();

  if (!response.ok) {
    throw new Error(
      `${response.status} ${body?.error?.code ?? "API_ERROR"} ` +
      `(request_id=${body?.request_id ?? "unknown"})`
    );
  }

  if (body.count !== body.data.length) {
    throw new Error("Unexpected pagination contract");
  }

  return body;
}

The client logs request_id in its error so a failed customer request can be traced without logging credentials or a full native payload.

Paginate comments without calculating offsets

async function collectComments(videoUrl, maximum = 50) {
  const comments = [];
  let cursor = 0;
  let hasMore = true;

  while (hasMore && comments.length < maximum) {
    const page = await getPage("/v1/videos/comments", {
      url: videoUrl,
      count: Math.min(10, maximum - comments.length),
      cursor,
      raw: false
    });

    comments.push(...page.data);
    cursor = page.cursor;
    hasMore = page.has_more === true;
  }

  return comments;
}

const comments = await collectComments(
  "https://www.tiktok.com/@sophiegamergirl/video/7658435999478992148"
);

console.log(comments.length);

The loop does not add 10 to the cursor. It uses the value returned by the successful response.

Use the same pattern for opaque cursors

async function* paginate(path, fixedParameters, pageSize = 20) {
  let cursor = 0;
  let hasMore = true;

  while (hasMore) {
    const page = await getPage(path, {
      ...fixedParameters,
      count: pageSize,
      cursor,
      raw: false
    });

    yield page;

    cursor = page.cursor;
    hasMore = page.has_more === true;
  }
}

for await (const page of paginate(
  "/v1/videos",
  { username: "sophiegamergirl" },
  20
)) {
  for (const video of page.data) {
    console.log(video.id, video.description);
  }
}

The same iterator works for:

/v1/music/videos
/v1/hashtag/videos
/v1/search/videos
/v1/search/users
/v1/search/hashtags
/v1/search/music
/v1/trending/videos

Merge native windows without losing raw data

A public upstream route may return fewer items than the client requested even when another native window exists. The service can fetch consecutive windows, normalize their records, remove duplicate IDs, and stop when one of these conditions is reached:

The requested client count is filled
TikTok reports no more data
A safe upstream work limit is reached

When raw=true is requested, the native responses are not discarded. They remain ordered under:

raw.pages[]

This keeps two different contracts:

Normalized fields are the stable interface for application code.
Raw pages expose the native upstream JSON when a product needs a field that is not normalized.

Raw mode should be deliberate. Native payloads are larger, bypass cache, and can change when TikTok changes its public web response.

Preserve `null` instead of inventing `false`

Public web surfaces do not expose every account, video, or comment field consistently. Booleans therefore need three states:

true: TikTok explicitly returned a positive signal
false: TikTok explicitly returned a negative signal
null: the selected upstream surface did not provide a trustworthy value

For example:

const researchRow = {
  id: video.id,
  is_ad: video.is_ad ?? null,
  is_ai_generated: video.is_ai_generated ?? null
};

Converting null to false would turn "unknown" into a factual claim.

Keep the infrastructure HTTP-first

Reliable pagination does not require one Chrome process per customer request.

The collection path can remain HTTP-first:

Use direct public HTTP surfaces when they return the required data.
Generate or refresh a web signature only for routes that require it.
Use one bounded browser fallback only when TikTok requires current browser state.
Return a structured 503 when the upstream is temporarily unavailable instead of fabricating an empty successful response.

This keeps idle memory low and avoids maintaining a persistent browser fleet.

Test the contract as a customer would

The production monitor for this API runs 29 customer-path probes through the RapidAPI gateway. It covers all 16 data endpoints with multiple creators, videos, search terms, hashtags, music, comments, replies, pagination, and empty final pages.

The checks validate behavior rather than only HTTP status:

count === data.length
No duplicate IDs inside a page
Returned identities match the requested resource
Empty reply pages are accepted when no public replies remain
Retryable upstream failures are not hidden inside a fake 200
Process memory, Swap, restarts, and endpoint latency remain observable

Try the complete API

The current release provides 16 read-only endpoints for public videos, creators, comments, replies, music, hashtags, search, Explore/trending data, batch lookup, and URL resolution.

You can run the examples in the RapidAPI playground:

https://rapidapi.com/pomiandaitumm/api/tiktok-public-metadata-api

Full documentation, runnable examples, limitations, and live status:

https://tiktok-public-metadata-api.pomiandaitumm.chatgpt.site

If you build cursor-based APIs, I would be interested in how you model continuation state: native offsets, opaque signed state, database keys, or another approach?

Why Xiaohongshu xsec_token Links Expire and How to Recover

mian po — Sun, 19 Jul 2026 11:24:16 +0000

Why Xiaohongshu xsec_token Links Expire and How to Recover

A Xiaohongshu note ID identifies a note, but it is not always sufficient request input. Public note links can also carry an xsec_token generated for that note and discovery context. Treat the complete link as short-lived input rather than a permanent identifier.

The Xiaohongshu URL Helper validates these components locally and creates a copy-ready request for Xiaohongshu (RedNote) Data API.

A usable note URL

The detail and comment endpoints expect a complete public URL:

https://www.xiaohongshu.com/explore/NOTE_ID
  ?xsec_token=CURRENT_TOKEN
  &xsec_source=pc_search

The token belongs to the link that supplied it. Do not invent one, move it between notes, or store a signed example as permanent application configuration.

Validate before fetching

Use the local parser first. It separates malformed input from the upstream request:

parsed = requests.get(
    f"{base_url}/note/parse_url",
    headers=headers,
    params={"url": note_url, "raw": "false"},
    timeout=30,
)
parsed.raise_for_status()
print(parsed.json()["data"]["note_id"])

Parsing proves that the URL shape is supported. It does not promise that an old token remains current upstream.

Recover instead of retrying forever

When the API reports that the note or token is no longer accessible:

Open the public note again from a visible card or freshly shared link.
Copy the complete browser address, including query parameters.
Validate the new URL locally.
Retry the detail request with the new URL.

For a temporary 503 REQUEST_TIMEOUT, keep the same URL, honor Retry-After, and retry the identical request. Token expiry and a temporary timeout are different failures.

Collect several current URLs

The URL Helper includes a bookmarklet that scans visible Xiaohongshu note-card links and copies up to 10 deduplicated URLs as batch JSON. It only reads link addresses. It does not read cookies, browser storage, API keys, or hidden page state, and it sends no network request.

Use the resulting JSON with POST /note/batch_detail_from_url or the asynchronous batch workflow.

Keep the RapidAPI key on the server

The helper never asks for a RapidAPI key. Keep that credential in a server-side environment variable and send it only to the RapidAPI gateway. Include data.request_id, not credentials or complete signed URLs, when asking for support.

Read the complete xsec_token guide or test the API with the free monthly allowance.

XHS Data API is independent and is not affiliated with or endorsed by Xiaohongshu/RedNote. Use public data lawfully and follow applicable law, third-party rights, and source-platform rules.

Process A Larger Xiaohongshu Note Batch Asynchronously

mian po — Sat, 18 Jul 2026 11:07:50 +0000

Process A Larger Note Batch Asynchronously

Synchronous batches are convenient for a small request. For up to 50 complete public note URLs, use the asynchronous workflow so the client does not hold one gateway request open.

Start with XHS Data API on RapidAPI and keep the RapidAPI key on your server.

Submit A Job

import os
import time
import requests

base_url = "https://xiaohongshu-rednote-data-api.p.rapidapi.com"
headers = {
    "X-RapidAPI-Key": os.environ["X_RAPIDAPI_KEY"],
    "X-RapidAPI-Host": "xiaohongshu-rednote-data-api.p.rapidapi.com",
    "Content-Type": "application/json",
}
urls = [os.environ["XHS_NOTE_URL"]]

response = requests.post(
    f"{base_url}/note/async_batch_detail_from_url",
    headers=headers,
    json={"urls": urls, "count": len(urls), "raw": False},
    timeout=30,
)
response.raise_for_status()
job = response.json()["data"]
print("job_id:", job["job_id"])

Poll At The Recommended Interval

while job["status"] in {"queued", "running"}:
    time.sleep(job.get("poll_after_seconds", 2))
    response = requests.get(
        f"{base_url}/note/async_batch_detail_result",
        headers=headers,
        params={"job_id": job["job_id"]},
        timeout=30,
    )
    response.raise_for_status()
    job = response.json()["data"]

print(job["status"], job["success"], job["failed"])

Stop polling when status is succeeded, partial, or failed. Results preserve per-item successes when another URL fails. Jobs process URLs sequentially, survive normal service restarts, and expire after 24 hours.

Never expose job_id as an authorization mechanism. Keep RapidAPI authentication server-side and store only the job metadata your application needs.

Download the complete clients and retry helpers from the developer website or review the public examples repository.

XHS Data API is independent and is not affiliated with or endorsed by Xiaohongshu/RedNote. Use public data lawfully and follow applicable law, third-party rights, and source-platform rules.

Retrieve Rich Xiaohongshu Note Data With Python

mian po — Sat, 18 Jul 2026 11:07:48 +0000

Retrieve Rich Xiaohongshu Note Data With Python

This tutorial retrieves normalized public note data from a complete Xiaohongshu (RedNote) URL through RapidAPI.

What You Need

A subscription to XHS Data API.
A server-side RapidAPI key.
A current public note URL containing the xsec_token copied with that note.

Keep both values in environment variables:

export X_RAPIDAPI_KEY="your_rapidapi_key"
export XHS_NOTE_URL="complete_current_public_note_url"

Validate The URL First

/note/parse_url performs local validation and extracts URL components without fetching upstream note data:

import os
import requests

base_url = "https://xiaohongshu-rednote-data-api.p.rapidapi.com"
headers = {
    "X-RapidAPI-Key": os.environ["X_RAPIDAPI_KEY"],
    "X-RapidAPI-Host": "xiaohongshu-rednote-data-api.p.rapidapi.com",
}

parsed = requests.get(
    f"{base_url}/note/parse_url",
    headers=headers,
    params={"url": os.environ["XHS_NOTE_URL"], "raw": "false"},
    timeout=30,
)
parsed.raise_for_status()
print(parsed.json()["data"]["note_id"])

Retrieve Normalized Detail

detail = requests.get(
    f"{base_url}/note/detail_from_url",
    headers=headers,
    params={"url": os.environ["XHS_NOTE_URL"], "raw": "false"},
    timeout=45,
)
detail.raise_for_status()
data = detail.json()["data"]
note = data["note"]

print("title:", note.get("title"))
print("author:", (note.get("author") or {}).get("nickname"))
print("type:", note.get("type"))
print("images:", len(note.get("images") or []))
print("request_id:", data["request_id"])

The normalized note contract can include author identity, description, engagement counts, tags, timestamps, location, images, video, music, and media dimensions when the public source supplies them.

Understand Missing Fields

null means the selected upstream page did not provide that field. It is not zero. Optional location, music, video, or engagement fields can legitimately be unavailable.

Media URLs may be signed or temporary. Use them promptly and do not treat them as permanent storage.

Handle Failures

Keep data.request_id from every error. A retryable 503 REQUEST_TIMEOUT may include Retry-After; wait for that interval before retrying the identical request. When a note token has expired, obtain a current complete public URL instead of inventing or transferring a token.

The complete Python and Node.js clients are available on the developer website and in the public examples repository.

XHS Data API is independent and is not affiliated with or endorsed by Xiaohongshu/RedNote. Use public data lawfully and follow applicable law, third-party rights, and source-platform rules.

Engineering Reliable Cursor Workflows for Public RedNote Data

mian po — Wed, 15 Jul 2026 09:12:39 +0000

Engineering Reliable Cursor Workflows for Public RedNote Data

Social-data integrations often fail in boring places: a copied page token expires, a cursor is reused with the wrong resource, a timeout is mistaken for an empty page, or a missing metric is silently converted into zero.

I built the public examples for Xiaohongshu (RedNote) Data API around those failure modes rather than around the happy path alone. The API exposes normalized public note details, top-level comments, verified author profiles, media inventories, and sequential batches. The BASIC plan includes 200 free requests each month for evaluation.

The complete Python and Node.js examples are available from the developer website as a versioned download.

Start With A Complete Public URL

Detail and comment requests use the complete public note URL, including its current xsec_token. Keep the URL and RapidAPI key in server-side environment variables:

export X_RAPIDAPI_KEY="your_rapidapi_key"
export XHS_NOTE_URL="complete_current_public_note_url"

First validate the URL locally:

parsed = client.parse_note_url(os.environ["XHS_NOTE_URL"])
detail = client.detail_from_url(os.environ["XHS_NOTE_URL"])
print(parsed["note_id"], detail["note"].get("title"))

The parser does not fetch upstream data. This separates malformed-input errors from upstream availability errors before the expensive step begins.

Treat Cursor State As A Contract

The first comment request has no cursor. Every later request must use the exact next_cursor returned by the previous page and keep the same note URL:

cursor = None
seen = set()

while True:
    page = client.comments_page(note_url, cursor=cursor, count=10)
    for comment in page["items"]:
        if comment["comment_id"] not in seen:
            seen.add(comment["comment_id"])
            yield comment
    if not page["has_more"]:
        break
    cursor = page["next_cursor"]

The client repository also deduplicates IDs defensively. A cursor is short-lived and resource-bound; editing it or moving it to another note is a client error.

A Timeout Is Not An Empty Page

When an upstream page cannot finish within the service budget, the API returns a structured error instead of 200 with an empty list:

{
  "code": 503,
  "msg": "Comments did not load",
  "error_code": "REQUEST_TIMEOUT",
  "data": {
    "request_id": "support-id",
    "retry_after": 3
  }
}

Wait for Retry-After, then retry the identical URL and cursor. Do not advance the chain. Long cooldowns are intentionally not hidden behind aggressive automatic retries.

Preserve Unknown Values

Public profile cards and note summaries do not always contain full engagement or media fields. The normalized contract returns null when the source did not supply a value. That is different from numeric zero and boolean false.

This detail matters in dashboards. Converting unknown likes to zero changes averages, rankings, and trend lines. Clients should preserve null until a richer detail response supplies the missing value.

Keep Support IDs, Not Secrets

Every response includes data.request_id. Log that ID with the endpoint and HTTP status. Do not log RapidAPI keys, cookies, browser state, or complete signed URLs.

Try The Examples

The repository includes:

a reusable Python client with structured errors and bounded retries;
Node.js 20 examples using built-in fetch;
duplicate-safe top-level comment pagination;
asynchronous batch submission and polling;
troubleshooting guidance for 400, 401, 403, 429, and 503 responses.

Start with the developer guides or download the client examples. The current public API focuses on stable read workflows and does not advertise note search, author-post feeds, or child-comment replies.

XHS Data API is independent and is not affiliated with or endorsed by Xiaohongshu/RedNote. Use public data lawfully and follow applicable law, third-party rights, and source-platform rules.

DEV Community: mian po

Engineering Reliable Pagination for Public TikTok Data

Define the response contract first

Numeric and opaque cursors are different

Build one reusable client

Paginate comments without calculating offsets

Use the same pattern for opaque cursors

Merge native windows without losing raw data

Preserve null instead of inventing false

Keep the infrastructure HTTP-first

Test the contract as a customer would

Try the complete API

Why Xiaohongshu xsec_token Links Expire and How to Recover

Why Xiaohongshu xsec_token Links Expire and How to Recover

A usable note URL

Validate before fetching

Recover instead of retrying forever

Collect several current URLs

Keep the RapidAPI key on the server

Process A Larger Xiaohongshu Note Batch Asynchronously

Process A Larger Note Batch Asynchronously

Submit A Job

Poll At The Recommended Interval

Retrieve Rich Xiaohongshu Note Data With Python

Retrieve Rich Xiaohongshu Note Data With Python

What You Need

Validate The URL First

Retrieve Normalized Detail

Understand Missing Fields

Handle Failures

Engineering Reliable Cursor Workflows for Public RedNote Data

Engineering Reliable Cursor Workflows for Public RedNote Data

Start With A Complete Public URL

Treat Cursor State As A Contract

A Timeout Is Not An Empty Page

Preserve Unknown Values

Keep Support IDs, Not Secrets

Try The Examples

Preserve `null` instead of inventing `false`