Coursera ships two big JSON blobs on every search page. Neither has the results.

#webscraping #python #html #api

Quick answer: Coursera's search page embeds both window.App and window.__APOLLO_STATE__, which look exactly like where the course data should live. Neither contains the search results. The 12 results per page are plain server-rendered HTML — div[data-testid="product-card-cds"] cards — and the two JSON blobs hold app/session bootstrap state and unrelated SEO footer links. If you go looking for a JSON payload here you will find two, read neither correctly, and conclude the page is client-rendered.

The reflex that costs you an afternoon

When a modern site's search page is ~750 KB, the reflex is: find the hydration blob, parse JSON, done. It is usually right. __NEXT_DATA__ on a Next.js site, __NUXT__ on Nuxt, an Apollo cache on a GraphQL front-end — pulling structured JSON out of a script tag beats writing CSS selectors that break when a designer changes a class name.

So the natural plan for Coursera is: grab __APOLLO_STATE__, walk the normalized cache, extract courses. Apollo's cache is exactly the sort of thing that holds a search result set.

Here it doesn't. We inspected both blobs rather than assuming, and what is actually in them is session bootstrap and footer links. The course cards never pass through either one.

Where the data actually is

Server-rendered in the HTML, which is the pleasant surprise:

div[data-testid="product-card-cds"]

Twelve per page, with the title, the partner, the rating, the review count and the URL all sitting in the markup. Pagination is a plain ?page=N query parameter — no cursor, no token, no continuation. You can fetch page 2 by editing the URL.

That is an easier target than the JSON path would have been. The point is not that Coursera is hard; it is that the assumption was wrong in the direction that wastes time. A blob that exists but doesn't contain what you want is worse than no blob at all, because you will keep digging through it convinced the data is in there somewhere.

The general lesson: inspect the blob, don't infer it

A hydration blob's presence tells you nothing about its contents. Three things can be true at once, and on Coursera all three are:

The page is server-rendered.
It ships large JSON state objects.
Those objects do not contain the rendered results.

The check costs one minute: pull the blob, dump its top-level keys, and look for a field that plausibly holds a list of results. If you can't find one, stop looking and read the HTML. We have made the opposite mistake before — treating a 200 response as proof of reachability without checking what was actually in the body — and it is the same error wearing different clothes: confirming the shape of a response instead of its contents.

One more thing worth knowing: there is no price on the page

If you are scraping Coursera hoping for a dollar figure, there isn't one. The search page never shows a literal price. The closest public signals are the access tags — Free trial, Build toward a degree, Job ready — which tell you the commercial tier without the number.

Similarly, the number next to a course is the review count, not enrollment. Those get conflated constantly in write-ups about Coursera data. 281,000 reviews on Python for Everybody is not 281,000 students; it's how many people rated it.

What a row looks like

{
  "query": "python",
  "title": "Python for Everybody",
  "url": "https://www.coursera.org/specializations/python",
  "partner": "University of Michigan",
  "product_type": "SPECIALIZATION",
  "difficulty_level": "Beginner",
  "duration": "3 - 6 Months",
  "rating": 4.8,
  "rating_count": 281000,
  "access_tags": ["Free trial", "Build toward a degree"]
}

product_type is derived structurally from the URL path rather than from a label, because the label is inconsistent and the path is not: /learn/ is a course, /specializations/ a Specialization, /professional-certificates/ a certificate, /degrees/ a degree. Structure beats copy when you need a field you can filter on.

🎓 We packaged this one: Coursera Courses Scraper turns Coursera search into clean, typed rows — title, partner, product type, difficulty, duration, rating, review count and access tags — paging as deep as you ask. $5.20 per 1,000 results, and you only pay for rows that land.

FAQ

Does Coursera's search page require JavaScript to get the results?
No. The 12 results per page are server-rendered into the HTML as product-card-cds cards. You do not need a browser.

Can I get course prices from Coursera search?
No literal price appears on the search page. The access tags (Free trial, Build toward a degree, Job ready) are the closest public tier signal.

Is rating_count the number of students enrolled?
No — it is the number of reviews. Enrollment is not exposed on the search page, and the two are widely confused.

How does pagination work?
A plain ?page=N query parameter. No cursor or continuation token, which makes deep paging straightforward — though you should still cap it, since each page is a few hundred KB.

DEV Community