DEV Community: Programming with Jack Chew

Reading links your agent can't open: Xiaohongshu, Douyin, TikTok, YouTube and X as an MCP tool

Programming with Jack Chew — Sat, 12 Sep 2026 02:56:14 +0000

Originally published at linkdigest.dev, where I build this.

The problem, in one paragraph

Paste a Xiaohongshu, Douyin, TikTok, YouTube or X link into an AI agent and it fetches the URL, gets an app-download shell or a login wall, and tells you there is nothing there. It is not wrong. The content of those posts is video, images, and text printed inside images, behind tokenised share links — none of it is in the HTML a fetch returns. Strip the <script> tags from a real Xiaohongshu note and 264 characters of navigation are left.

What LinkDigest does about it

LinkDigest is a hosted reader: one call turns the link into text an LLM can use — a transcript with timecodes, the on-screen text, a description and OCR of every image, the caption and metadata — as Markdown or JSON. It is an MCP server (claude mcp add --transport http linkdigest https://linkdigest.dev/mcp --header "Authorization: Bearer $KEY"), a REST API, and a web console. Measured on a 17-image Xiaohongshu note: 17 images described and read, 381 on-screen text fragments, 13 key points, 119 seconds. Three digests are free, no card.

What it does not do, stated up front: Bilibili refuses our server's address (HTTP 412), Instagram is wired but not verified, Facebook is out of scope. A digest that could not read part of a post says so in a degraded field instead of returning a thin result quietly — and a digest that read nothing costs nothing.

Your agent is good at reading. It just can't open half the internet.

Hand Claude Code a Xiaohongshu link from a design review, a Douyin video a colleague sent, a TikTok your competitor posted — and it fetches the URL, gets an app-download shell, and tells you it couldn't find anything useful. It isn't wrong. There genuinely is nothing there.

[Diagram: The same URL returns an app-download shell to a plain fetch, and the actual note to LinkDigest.]

This is a post about the second box: how to call it, and three things people actually do with it.

The agent calls it, not you

One command:

claude mcp add --transport http linkdigest \
  https://linkdigest.dev/mcp \
  --header "Authorization: Bearer ld_live_..."

That registers one tool, digest_url(url, format, job_id). Only url is required.

The part that matters is what happens next: you never mention it again. The tool description tells the agent to call it whenever it meets a social link it cannot read, so the agent reaches for it on its own, mid-task, the way it reaches for grep.

[Diagram: The agent calls the MCP server, which calls the engine, and the transcript returns to the agent.]

Cursor takes the same server in ~/.cursor/mcp.json:

{
  "mcpServers": {
    "linkdigest": {
      "url": "https://linkdigest.dev/mcp",
      "headers": { "Authorization": "Bearer ld_live_..." }
    }
  }
}

What actually comes back

Not a summary. The parts:

curl -X POST https://linkdigest.dev/api/v1/digest \
  -H "Authorization: Bearer ld_live_..." \
  -H "Content-Type: application/json" \
  -d '{"url": "https://v.douyin.com/..."}'

A real response, trimmed — this is a Douyin video, run against production while writing this paragraph:

platform           'douyin'
author             '逸蒙的赛博空间'
posted_at          '2026-08-27'
transcript_source  'asr'
cached             True   credits 0
degraded           []
transcript[0]      {"t": 0, "text": "谁能想到，就这么一个丑萌小玩意儿…"}
key_points[0]      "该AI产品名为Tolen，主打「记住你」的核心功能…"

Fifteen fields in total: platform, author, title, posted_at, caption, transcript, ocr_text, images, key_points, raw_markdown, source_url, transcript_source, degraded, plus cached and credits.

Two of those are worth pointing at.

transcript_source says how you got the words — native_captions, asr, or none. Captions the platform already had are exact; speech recognition is not. A pipeline that treats those identically will eventually quote a mis-heard number back at someone as fact.

degraded is a list of what didn't fully work, in plain words. Empty on a clean run. It exists because of a specific bug: a note was digested during a provider rate-limit storm, every vision batch failed, the per-batch handler swallowed each one, and the job returned images: 0, ocr: 0 — a well-formed, completely empty result, which was then cached for thirty days. Nothing downstream could tell an empty answer from an easy one. Now it can.

The pipeline, and why some links take two calls

[Diagram: Four stages: resolve, fetch, read, structure.]

read is where the time goes, because it is the only stage that has to watch or listen to anything. Measured: a Xiaohongshu note with images takes one to two minutes; a YouTube video with captions about two and a half.

That is longer than one HTTP request should wait, so the API hands the work back:

POST /api/v1/digest
→ 202 {"pending":true,"jobId":"abc…","poll":"/api/v1/digest/abc…","retryAfter":15}

Collect it with the same key at that poll path — 202 while it runs, 200 with the digest when it's done. Over MCP you don't do any of this by hand: the tool tells the agent to call digest_url again with the job_id and no url, and the agent does.

Three things people build with it

1. The agent that reads the link itself

The one that needs no code. A teammate drops a Douyin link in an issue; the agent working that issue reads it without anyone transcribing anything. This is the whole reason the tool description is written the way it is — it is instructions for a machine deciding whether to reach for a tool, not marketing copy.

2. A list of links becomes a table

Fifty URLs in, fifty rows out — one POST each, or the Apify Actor if you'd rather not write the loop. The dataset has a row per link with the transcript, the on-screen text and the key points already separated.

Worth knowing what you're joining: there is real demand for exactly this shape of work. A single competing Douyin scraper on the Apify Store has over 1.5 million runs. Almost all of them return metadata — view counts, captions, author handles. Very few return what was actually said.

3. Watching what a competitor publishes

The same loop on a schedule. A creator's video output becomes searchable text over time, so "when did they first mention pricing" is a grep instead of an afternoon of watching.

This is the use case where transcript_source earns its place again. If you're going to quote a competitor's claim back to your own team, it matters whether a human captioned it or a model guessed at it.

What it costs, and why it isn't flat

A digest is not a unit of cost. Measured against real runs:

[Diagram: Measured cost per digest: a web article is a fraction of a cent, a nineteen-minute video is nine cents.]

Which is why pricing counts the two things that drive that spread — images to describe, minutes to transcribe — rather than counting requests. One credit covers a typical post and its first six images; each further six images adds one, and each minute of media adds two.

Cached links are free and never counted. Anything anyone has ever digested comes back in about a second, for nobody's credits.

Where it stops

The honest table, because finding out after you've wired something in is worse than knowing now:

Works: Xiaohongshu, Douyin, TikTok, YouTube, X, ordinary web pages.
Bilibili returns HTTP 412 to a datacenter address. Needs a residential proxy.
Instagram and Facebook need a logged-in session for most posts.

Three free digests, no card, if you want to check any of that yourself.

The platform-by-platform detail — what each one returns and what breaks — is in What it actually takes to read a Xiaohongshu post from a server, and per platform on the coverage pages.

What it actually takes to read a Xiaohongshu post from a server

Programming with Jack Chew — Sat, 12 Sep 2026 02:53:41 +0000

Originally published at linkdigest.dev, where I build this.

The problem, in one paragraph

What LinkDigest does about it

Every claim here was checked against a live link. Where something doesn't work, it says so.

Paste a Xiaohongshu link into an AI coding agent and it sees nothing. Same for Douyin. The usual answer — "just use yt-dlp" — is half true in a way that wastes an afternoon, so here is the whole picture.

yt-dlp only reads half of Xiaohongshu

XiaoHongShuIE reads note.video.media.stream. That is a video note. Image notes — 图文, a caption plus a stack of photos — are the majority of the platform and usually the ones worth reading, and the extractor has nothing to say about them.

Getting those means parsing the note payload out of the page itself: the state blob the page ships, and the imageList inside it. Reusing yt-dlp's own js_to_json and traverse_obj for that keeps you aligned with upstream when the page shape shifts, which it does.

The page is different depending on who you say you are

This one cost me an embarrassing amount of time.

Request a note with a mobile user agent and you get 202KB of app-download shell whose <title> is just the site name. Request the same URL with a desktop user agent and you get 85KB containing the real note.

Both are HTTP 200. Nothing tells you that you got the wrong one except that the content isn't there.

[Diagram: The same note URL returns 202KB of app shell to a mobile user agent, and 85KB containing the note to a desktop one.]

Cold requests are rejected

Fetching a note URL directly, with no prior session, does not work. Fetch the /explore feed first, keep the cookie jar it gives you — acw_tc, abRequestId — and then the note loads.

No account, no credentials, no API key. Just the same two-step a browser performs without you noticing.

Douyin does not negotiate

Coverage and output for Douyin: platforms/douyin.

Douyin refuses anonymous requests outright: captcha on the web page, 403 from the APIs. There is no user-agent trick here. That path needs cookies from a logged-in session, and yt-dlp does not support the platform at all.

YouTube blocks your server, not your laptop

Coverage and output for YouTube: platforms/youtube.

The single most common "it worked locally and broke in production" report. YouTube blocks datacenter IP ranges, so a yt-dlp fetch that is perfect on your machine fails from EC2, App Runner, Lambda or anywhere else you deploy — and no amount of configuration fixes an IP-range block.

The workable fallback is a model that watches the video and returns a transcript, which costs meaningfully more than parsing captions and is worth measuring separately.

Where it stops

Being honest about this saves everyone time:

Bilibili returns HTTP 412 to a datacenter address. Needs a residential proxy.
Instagram and Facebook need a logged-in session for most posts.
TikTok resolves fine but rate-limits under load; a frame fetch can come back 403 mid-job.

The part nobody mentions: failure has to be loud

The bug that taught me the most had nothing to do with extraction.

A note was digested during a provider rate-limit storm. Every vision batch failed, the per-batch error handler swallowed each one, and the job returned images: 0, ocr: 0 — a perfectly well-formed, completely empty result. It was then written to a shared cache with a 30-day TTL.

Long after the underlying problem was fixed, that link still returned nothing, because a cached answer is cheaper to serve than to recompute. The pipeline was healthy. The cache was serving a fossil, and nothing downstream could tell the difference between an empty result and an easy one.

Two things came out of that:

A step that loses content has to say so. The result now carries what failed, so a thin answer is distinguishable from a genuinely short post.
Thin results get a short TTL, not the full one. Not "don't cache" — a link that fails every time would then re-run the full paid pipeline on every request forever. An hour bounds the staleness and the spend together.

If you are building anything that caches derived content, that second one is the trap. The obvious fix is the expensive one.

The findings are the same whether you use a hosted service or write your own.

For how this fits into an agent's workflow end to end — the MCP tool, the job id for long media, what the output looks like — see Reading links your agent can't open.

LinkDigest: turn Xiaohongshu, Douyin, TikTok, YouTube and X links into text your agent can read

Programming with Jack Chew — Sat, 12 Sep 2026 02:42:47 +0000

I build LinkDigest. Every number below is from a measured run recorded in the project log; nothing is projected.

The problem

Agents are good at reading and bad at opening. Give Claude Code, Cursor, or a Dify workflow a link from Xiaohongshu, Douyin, TikTok, YouTube or X and the usual tool — fetch the URL, parse the HTML — returns one of three things: an app-download shell, a login wall, or a page whose body is 264 characters of navigation because the post itself lives in a JSON blob or in a video. The agent reports "nothing useful here" and it is telling the truth about the HTML.

The substance of these posts is not text. It is speech in a video, words printed on images, screenshots of code, a recipe laid out in a photo. A plain fetch throws all of that away, and a share link from the app carries a token that expires in a few weeks, so even the HTML you do get is different depending on which link and which user-agent you used.

The solution

LinkDigest does the reading on its own servers and hands back text. One call:

curl -X POST https://linkdigest.dev/api/v1/digest \
  -H "Authorization: Bearer $LINKDIGEST_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://www.youtube.com/watch?v=CEvIs9y1uog", "format": "json"}'

returns the post's transcript with timecodes, every fragment of on-screen text, a description and OCR of each image, the caption and metadata, and a list of key points — as JSON, or as Markdown if you ask for it. Long media answers in two steps: a 202 with a job id, then the digest when it is done.

Measured, not estimated:

post	what came back	time
a 17-image Xiaohongshu note	17 images described and read, 381 on-screen text fragments, 13 key points	119 s
a Xiaohongshu video note	ASR transcript, 56 on-screen text fragments, 12 key points	~4 min
a YouTube talk (native captions)	426 transcript segments, 190 on-screen text fragments	~1 s cached
a two-photo X post	both photos described, 3 text fragments	~20 s

For agents: it is an MCP server

claude mcp add --transport http linkdigest https://linkdigest.dev/mcp \
  --header "Authorization: Bearer $LINKDIGEST_KEY"

One tool, digest_url(url, format). The tool description tells the model exactly when to reach for it and which platforms are out, so an agent does not confidently hand you a failure. It is in the official MCP registry as dev.linkdigest/linkdigest, on Apify as three Actors, and submitted to the Dify marketplace.

What it does not do

This is the part I would want to know before pasting a key:

Bilibili: not supported. Bilibili returns HTTP 412 to our server's address.
Instagram: wired, not verified end to end. Facebook: out of scope.
X long-form articles expose only their lead image; regular photo posts come back in full.
A YouTube link is read by Gemini watching the video, because YouTube blocks datacenter IPs; if Google refuses a video, you get an honest stub, and it costs nothing.
Anything that could not be read is named in a degraded field. A digest that read nothing is free.

Pricing

Three digests free, no card. Then $9 a month for 500 credits — a typical post is 1 credit, video 2 per started minute, and links anyone has already digested are free forever. https://linkdigest.dev/pricing

Why I built it

I kept pasting Xiaohongshu and Douyin links into agents and watching them fail, and the failures were silent — a confident summary of a login page. Two of the bugs I hit while building this are written up separately: share links have two domains and one of them fails silently, and what it actually takes to read a Xiaohongshu post from a server.

If you try it and something reads thin, reply here — I fix these the same day.

Xiaohongshu share links have two domains, and one of them will fail silently

Programming with Jack Chew — Sat, 12 Sep 2026 02:33:42 +0000

Originally published at linkdigest.dev, where I build this.

Every number here came from a live link on 2026-09-08 and 2026-09-10. The failure described is one this service shipped to a real user.

If you have a host table that maps domains to platforms, and Xiaohongshu is in it, check what you wrote. There is a good chance it says xhslink.com and nothing else. That was true here, and it cost us the first person who ever signed up.

Two domains, same platform

Share a note from the Xiaohongshu iOS app today and the clipboard gets something like:

https://xhslink.cn/o/AxnRePgIokn

Older shares, and links that travel through the desktop site, use .com:

https://xhslink.com/o/1WiQ1QI6Uc0

Both are real. Both resolve to the same place. Following the .cn one:

$ curl -sIL 'https://xhslink.cn/o/AxnRePgIokn' -A '<desktop UA>'
...
https://www.xiaohongshu.com/discovery/item/6a9d0eb2000000002a024480
    ?app_platform=ios&app_version=9.45&xsec_source=app_share
    &type=normal&xsec_token=CB7rCz...&share_id=310c7d81...

The second link we tested carried type=video instead of type=normal, which matters later.

Why the miss is invisible

A URL router usually looks like this, and ours did:

(re.compile(r"(^|\.)(xiaohongshu\.com|xhslink\.com)$"), Platform.xiaohongshu),

A .cn share does not match. In most systems it does not raise — it falls through to whatever generic handler sits at the end of the chain. Ours fetches the page and extracts what it can from the HTML.

And that works, in the sense that it returns 200. Xiaohongshu's page ships a <title> and a meta description containing the whole caption, so a generic extractor comes back with a title, an author of "小红书", and the caption. Enough to look like an answer.

It is not, and the reason is covered at length in what it actually takes to read a Xiaohongshu post from a server: strip the <script> tags from that page and 264 characters of navigation chrome are left. The note body is not in the rendered DOM at all. A generic extractor is reading the wrapper, not the post.

Here is what the same link returned before and after the domain was added to the table:

	`.cn` unrecognised	`.cn` recognised
platform	`web`	`xiaohongshu`
images described	0	1
on-screen text fragments	0	4
transcript	0	0
author	`小红书`	the actual account name
posted date	empty	`2026-09-06`
`degraded`	`["extracted from page HTML only — no media, transcript, or platform metadata"]`	`[]`

And the second link, the type=video one:

	before	after
platform	`web`	`xiaohongshu`
transcript source	none	`asr`
on-screen text fragments	0	56

A transcript existed the whole time. Nothing errored. The request succeeded in seven seconds and returned a post with no transcript in it.

That is the part worth taking away. A silent downgrade is worse than a crash, because a crash tells the user to retry and a downgrade tells them this is what your product can do. The person who hit this did not file a bug. They opened their usage page, their settings, the docs, and left.

Check your own table

If you maintain anything that classifies Xiaohongshu URLs, the fix is one character class:

(re.compile(r"(^|\.)(xiaohongshu\.com|xhslink\.(com|cn))$"), Platform.xiaohongshu),

And separately, if you expand short links before identifying content — you should, because the note id is not in the short form — xhslink.cn has to be in that set too. Ours was missing from both.

The reason a full test suite did not catch it is worth saying out loud: every Xiaohongshu URL in our fixtures, evals, docs and demo scripts was a .com one, because those were the links we had pasted. The test suite was green and tested the wrong domain.

The second half: your cache may outlive your fix

We deployed the routing fix and the very next request came back wrong anyway.

The cache key was computed from the raw URL by the layer in front, which had already stored the bad result under the key for xhslink.cn/o/AxnRePgIokn. The lookup answered before the fixed code ran. Deploying a parser fix does not repair anything you have already cached under the same key.

If you cache digests, decide deliberately which layer expands short links. Expanding before the key is computed also means two share links to the same note stop producing two entries, and stop billing twice.

What this service does now

xhslink.cn and xhslink.com both resolve to the Xiaohongshu path, image notes and video notes both, and a link that cannot be read fully says so in degraded rather than returning a thin result quietly.

Still not supported, stated where you can see it before paying: Bilibili returns HTTP 412 to our address, and Instagram is wired but not verified end to end.

If you are wiring this into an agent rather than calling it by hand, reading links your agent can't open covers the tool call and what each field contains.

Three digests are free and there is no card. linkdigest.dev

2026 Q1 is the year developers still build the agent harness. 2026 Q3 / 2027 is the year the LLM builds its own harness.

Programming with Jack Chew — Sun, 24 May 2026 03:12:16 +0000

2026 Q1 is the year developers still build the agent harness.

2026 Q3 / 2027 is the year the LLM builds its own harness.

Today, every AI coding agent — Claude Code, Cursor, Codex, Gemini CLI, Aider, you name it — depends on the same hidden layer:

the files that brief the agent before it starts work.

AGENTS.md
CLAUDE.md
.cursor/rules
SKILLS/
MCP server lists
memory schemas
test commands
lint commands
“Do not touch these paths.”
“Require human approval before this.”

Different IDE, same boilerplate.
Different repo, same boilerplate.
Different agent, same boilerplate.

That is the agent harness problem.

The hidden work behind AI coding agents

Most people talk about the coding agent itself.

But in practice, the quality of an AI coding session often depends on the context layer around the agent.

Before the agent starts coding, it needs to know:

what kind of project this is
what framework it uses
what files are important
what commands run tests
what commands run linting
what paths should not be touched
what tools are available
what memory should persist
what failure modes to avoid
what coding conventions to follow
when human approval is required

Without this layer, even strong coding agents can make subtle mistakes.

With this layer, the same agent can behave much more consistently.

That layer is what I call the harness.

Why this still exists in 2026

In theory, the LLM should be able to inspect a repo and generate all of this itself.

In practice, we are not fully there yet.

The models are smart enough to do real coding work, but not always reliable enough to deterministically generate perfect project-specific ground truth from scratch on every fresh repo, every time.

They can do it sometimes.

Not always.

So the human stays in the loop.

We write the same repo instructions again.

We copy the same rules across projects.

We maintain separate files for Claude Code, Cursor, Codex-style agents, Continue, Windsurf, and others.

Small work per repo.

Painful in aggregate.

The future: self-generating harnesses

I think this is temporary.

Soon, the coding model should be able to:

read the repo
understand the task
detect the project type
generate the right harness
connect the right tools
create memory schemas
write validation scripts
refine the loop until the task is complete

At that point, the harness layer disappears as a separately authored artifact.

But until then, developers still need a bridge.

I built harnessforge

I built harnessforge to test this idea.

It is a local, open-source harness generator for AI coding agents.

It is not another coding agent.

Your coding agent stays the brain.

harnessforge just lays down the ground truth the agent reads before work begins.

Run:

uvx harnessforge init

or install:

pip install harnessforge

In a few seconds, fully local with no network calls by default, it inspects your repo and generates startup files commonly used by AI coding agents.

What it generates

Depending on the project and blueprint, harnessforge can generate files such as:

AGENTS.md
SOUL.md
TOOLS.md
MEMORY.md
SKILLS/
.claude/CLAUDE.md
.cursor/rules
.continue/
.windsurf/rules
blueprint-specific validators

The goal is simple:

give the coding agent a stronger starting point.

Current blueprints

The current version includes these blueprints:

`rag-agent`

For retrieval systems, knowledge-base agents, citation enforcement, and grounded responses.

`finance-agent`

For finance or stock-related agents, including market-data handling and validation rules around trade execution safety.

`support-agent`

For customer support flows such as intent detection, knowledge-base lookup, ticket creation, escalation, and ticket lineage.

`workflow-agent`

For multi-step orchestration with tool logs, idempotency, and validation structure.

`python-cli-app`

A default blueprint for greenfield Python CLI projects.

Why this matters

The important idea is not the specific files.

The important idea is that coding agents need a reliable project-specific operating context.

Today, we manually maintain that context.

Tomorrow, the model may generate it automatically.

harnessforge is meant to sit in the middle.

A bridge, not a moat.

Use it now.

Throw it away when the models catch up.

Example workflow

uvx harnessforge init

Then open Claude Code, Cursor, Codex, Gemini CLI, Aider, or another coding agent inside the repo.

The agent now has project-specific context files to read before it starts work.

Instead of starting from a blank repo, the agent starts with:

project rules
tool definitions
memory structure
validation expectations
blueprint-specific failure modes
agent-specific startup files

The coding agent still writes the code.

The harness just gives it the right context.

The bet

My bet is:

2026 Q1: developers still build the agent harness.

2026 Q3 / 2027: the LLM builds its own harness.

Until that happens, a local deterministic harness generator can make AI coding workflows more reliable.

GitHub:
https://github.com/jcaiagent7143-ui/harnessforge

PyPI:
https://pypi.org/project/harnessforge/

I would love feedback from developers using Claude Code, Cursor, Codex, Gemini CLI, Aider, Continue, Windsurf, or other coding agents in real repos.

How are you managing your agent harness today?

Are you manually maintaining AGENTS.md, CLAUDE.md, .cursor/rules, MCP configs, memory files, and validation rules?

Or do you think the next generation of coding models will generate this layer automatically?