Page One Gave Me 537 Working WeChat Links. Page Two Gave Me 101 Out of 391

I run a visa agency in Bali, and the side of my work nobody asks about is scrapers: I maintain a shelf of Apify Actors, mostly because I needed the data myself first. One of them reads WeChat Official Account articles — WeChat Articles Scraper. WeChat is where Chinese brands, trade media and local dealers publish, and it is closed to the open web: no public search, no feed to subscribe to, nothing to crawl. The way in is Sogou Weixin search, the one public index of WeChat articles.

That makes every row a two-request affair, and this is the whole point of this post:

1. GET weixin.sogou.com/weixin?type=2&query=比亚迪&page=1 — a search page with ten results. Title, summary, account name, a cover image, a timestamp.
2. GET weixin.sogou.com/link?url=…&k=…&h=… — a tiny page whose only job is to assemble the real mp.weixin.qq.com URL out of string fragments inside a <script>.

Step 1 gives you metadata. Step 2 gives you the article. And they do not fail together.

949 rows, all fine, except the field that matters

I had 28 runs in the Actor's history between 6 and 31 October 2026 — health checks, buyer runs, my own tests. Every one finished SUCCEEDED, for 949 article rows, each with a title, a summary, an account name and a timestamp.

Then I counted articleUrl:

rows                    949
articleUrl present      659
articleUrl null         290   (30.6%)

by search page:
  page 1    537 rows      0 null
  page 2    391 rows    290 null   (74%)
  page 3     21 rows      0 null

Per slot inside a keyword the cliff is sharp: position 11 lost 3 of 37, position 13 lost 29 of 41, and from 15 onward roughly nine of ten rows had no link to the article they describe.

Nothing on the platform tells you this. The run is green, the dataset is full, the log is quiet — the Actor caught the failure, left the field null and moved on.

The thing that breaks is not the search

My first guess was pacing: a slow run on 11 October that crawled at 3.98 s per row resolved 38 of 40 links, while the fast runs (0.5 s per row) resolved about 26 of 40. So I rebuilt the two requests in 30 lines of Python and measured from outside the Actor.

One keyword, two pages, 0.3 s between link resolutions, straight from my own server:

page 1: http 200  31792B  links 10   cookies ABTEST SNUID SUID
  p1 #1..#10   http 200  822B  OK
page 2: http 200  32096B  links 10
  p2 #1..#10   http 200  822B  OK
RESULT delay=0.3s resolved 20/20

Twenty for twenty, page two included. Same test through an Apify datacenter proxy, which is what the Actor uses: 20/20 again. So neither pacing nor cloud IPs explain it.

Then I ran the two keywords the Actor's input prefill uses — 比亚迪 and 人工智能 — at the same time, from the same address, which is exactly what the Actor does with its default concurrency: 2:

人工智能   p2 #4  http 200    822B  OK
人工智能   p2 #5  http 200   5712B  NULL
比亚迪     p2 #4  http 200    822B  OK
比亚迪     p2 #5  http 200   5703B  NULL
RESULT 14/20  and  14/20

Both sessions flipped in the same second, after 28 resolutions between them. The giveaway is the body size: a working link page is 822 bytes, the failing one is 5.7 KB. Inside the big one, stripped of markup:

搜狗搜索   IP：2.29.62.170   访问时间：2026.11.01 13:14:34
VerifyCode：0abae66177f0   From：weixin.sogou.com
此验证码用于确认这些请求是您的正常行为而不是自动程序发出的

A CAPTCHA page, served with HTTP 200, with my exit IP printed on it. Its <title> is just 搜狗搜索 — "Sogou Search" — so title-sniffing does not catch it either. And the budget is specific to /link: for a good while after that, plain search pages kept coming back at 32 KB with ten results each, so a scraper that checks only its search requests sees a perfectly healthy site. (Keep hammering and the search page goes too — mine answered a 5,378-byte CAPTCHA to every search for twelve minutes straight after my experiments.)

A fresh IP is not enough, and that was the actual bug

The obvious fix is rotating to a new address. I tested that before writing it, because Sogou ties link resolution to the SNUID cookie it hands out with the search page, and a link issued to session A might be unusable anywhere else. Three attempts on one sogouLink:

A. throttled IP, old session                    200   5702B   NULL
B. fresh IP, new cookie jar, no search page yet  200   5707B   NULL
C. same fresh IP, after loading the same search page   200    822B   OK

B is the interesting line: a brand-new address, zero history, still a CAPTCHA — because the request arrives with no SNUID. Load one search page on the new session and the very same link resolves. The link belongs to the search result; the permission to resolve it belongs to whoever holds a current cookie. Rotation has to carry a warm-up request, or it does nothing.

That turned into about fifteen lines in the Actor:

async function resolveWithRotation(state, it, searchUrl) {
    for (let t = 0; t < 3; t++) {
        try {
            const url = await resolveLink(state.session, it.sogouLink, searchUrl);
            if (url) return url;
        } catch (err) { /* verification page — fall through to rotation */ }
        if (t === 2) break;
        // fresh address + its own SNUID: load the same search page, then try again
        state.session = await newSession(t + 2);
        const warm = await state.session.client.fetch(searchUrl, { headers: HEAD });
        await warm.text();
        await sleep(400 + Math.random() * 400);
    }
    return null;
}

Before and after, same input, same hour, 1 November 2026:

build 0.1.29 (old)   40 rows   15 with articleUrl null   21 s
build 0.1.30 (new)   40 rows    0 with articleUrl null   35 s   (2 rotations)

Two rotations recovered fifteen rows. A bigger confirmation run with full text on: 40 rows, 0 missing links, 38 with article text, 150 s. Rows still unresolved after three addresses now carry an articleUrlNote instead of a silent null — write the reason into the row, because "field is empty" and "field is empty because we were CAPTCHA'd" are different bugs you cannot tell apart a month later.

The link you worked so hard for expires

articleUrl is not a permalink. It looks like this:

https://mp.weixin.qq.com/s?src=11&timestamp=1793510361&ver=7061&signature=…&new=1

timestamp is not the article's publish time — it is the moment the search session signed the link, identical for every row of the same run. I pulled five links out of old datasets (two from 6 October, one from 21 October, two from a run eight hours earlier) and fetched them:

HTTP 200   27,951 bytes   链接已过期   ("the link has expired")

All five, byte-identical. A link from a fresh run, by contrast, returns 3.5 MB of real article with var msg_title in it — and was still alive on every poll for the next 12 minutes. So the practical rule: 200 OK means nothing here. If you store articleUrl for later, you are storing a 28 KB expiry notice. Either open it immediately, or switch on the full-text option so contentText, author, accountId and the image URLs are saved into the row while the link is still valid. A cheap guard in your own pipeline:

r = requests.get(row["articleUrl"], timeout=30)
if "链接已过期" in r.text or len(r.content) < 50_000:
    row["articleStatus"] = "link expired"   # not an HTTP error, just a dead end

What this index actually is: today, not history

One more thing fell out of the 949-row ledger — the gap between each article's timestamp and the moment it was scraped:

age at scrape:  median 2.2 h   p90 10.1 h   max 18.3 h
under 24 h:     949 of 949

Not one article older than 19 hours, across 26 days of runs. Sogou's keyword index is a recency feed, not an archive: twenty results per keyword means today's twenty — perfect for monitoring on a schedule, useless for last quarter's coverage. Nor is it a media list. Those 949 rows came from 800 distinct accounts, 692 of them appearing exactly once, and the most frequent publisher on 比亚迪 was not a newspaper but a single BYD dealership branch in Linfen, posting nine times.

Two more fields deserve suspicion, both measured on 38 rows that carry both the search version and the article-page version:

• accountName is a display string from the index. It disagreed with the name on the article page on 5 of 38 rows — 志言质语 vs 言质有锂, 杭州潮生活 vs 余杭潮生活, 小王科研 vs 博识网001. Accounts rename themselves; the index keeps the old label. If you are grouping by publisher, group on accountId (gh_522bcfbb1636), which comes from the page.
• publishedAt is closer to "indexed at". Median gap to the page's own publishedAtExact: 16 seconds. Maximum gap in the same 36 rows: 19.7 days — a research-grant notice the search dated 1 November at 03:50 and the article page dated 12 October at 10:00, because the account re-pushed it. Fifteen of 36 page timestamps end at :00 seconds, so the page value is minute-precision; treat the search value as the index event and the page value as the publish minute.

Running it, and when not to

One call, rows in the response:

curl -s -X POST "https://api.apify.com/v2/acts/lergassy~wechat-articles-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"keywords":["比亚迪","固态电池"],"maxArticlesPerKeyword":20,"includeContent":true}'

Rows cost $0.003 each, full text $0.005 more, and in the confirmation run two articles whose body timed out came back with contentError and were not charged for text. One row in the output is {"type":"notice"} with the run's own cost summary — filter on type != "notice".

Where this Actor is the wrong tool: no read counts and no likes, because WeChat shows those only inside the app; no comments; no account back catalogue, only what keyword search surfaces today, capped at 100 rows per keyword (ten pages of ten); Chinese keywords work far better than English ones; and two of forty rows in my last run were image posts — 20 characters of text and 28 photos — a real WeChat format, not an error.

What I will keep from this one, though, is cheaper than any of it: count the field that cannot be faked. Titles, summaries and account names arrive from the search page whatever happens, so a dataset full of them says nothing. The link is the only field that requires the site to say yes twice — and it was null on 290 rows while every status code on the platform said the job went fine.