DEV Community: CleanData Labs

Get an alert when a competitor launches a new Google ad (n8n, no code, pay only for new ads)

CleanData Labs — Fri, 30 Oct 2026 13:11:53 +0000

You want to know when a competitor starts a new Google campaign: a new search ad, a new banner, a new YouTube spot. Google shows all of this publicly in the Ads Transparency Center. The catch is that someone has to open it every few days, find the advertiser and scroll through hundreds of creatives to spot what changed.

This guide sets up a small n8n workflow that does it for you. Every morning it checks your competitors and posts only the ads that are new since yesterday to Slack. On quiet days it posts nothing and costs about a tenth of a cent.

The honest answer first: there's no official API

If you searched for "Google Ads Transparency Center API", here's what you'll find:

There's no public API for the Transparency Center. The Google Ads API support team said so on the official forum: "there is no API endpoint available for the Google Ads Transparency Centre". The Google Ads API only covers your own accounts.
The BigQuery dataset is partial. Google publishes bigquery-public-data.google_ads_transparency_center (tables creative_stats and removed_creative_stats). It's free to query within BigQuery's free tier and great for market-wide SQL. But it only covers ads served in the European Economic Area and Turkey. Ads shown only in the US, UK, Canada or Australia aren't in it. That's why this Stack Overflow question finds 3 creatives in BigQuery for an advertiser that shows about 200 on the website.

So if your competitors advertise in the EEA and you're comfortable with SQL, start with BigQuery. For anything else, including "tell me when something new shows up", you have to read the public website itself. That's what the rest of this post does.

What we'll build

Every day at 08:00
  â†’ Set competitors and alert settings   (the only node you edit)
  â†’ Fetch new competitor ads             (HTTP call to an Apify Actor, onlyNew = true)
  â†’ Build Slack summary                  (one message, ad text + links)
  â†’ Anything new to report?              (no â†’ stop, nothing posted)
  â†’ Post new ads to Slack
  â†³ on error: Post failure notice to Slack

The scraping is done by Google Ads Transparency Scraper, an Actor on the Apify platform. Disclosure: we built it (CleanData Labs), and this tutorial is partly there to show it off. Other Actors and APIs read the Transparency Center too (SerpApi and SearchApi have endpoints for it, for example). The n8n side works the same with any of them as long as you can ask for "only new" results, or keep track of creative IDs yourself.

We use it here because of one feature: with onlyNew turned on, the Actor remembers which ad IDs it has already returned and skips them. You're only charged for ads you haven't seen before. So the daily check stays cheap and you don't need a database in n8n to compute the diff.

"No code" is mostly true. There's one Code node that formats the Slack message, but it's ready to use and you don't have to touch it.

You need: an n8n instance (self-hosted or n8n Cloud), a free Apify account and a Slack workspace.

Step 1: find your competitor's advertiser (the `AR...` ID)

Every advertiser account in the Transparency Center has an ID that starts with AR, like AR17410177287600472065. You don't strictly need it, because the workflow can search by website. But it helps to know who you're actually watching, since one website often has several advertiser accounts.

By hand: open adstransparency.google.com, type the brand or website into the search box and pick the advertiser. The ID is in the URL:

https://adstransparency.google.com/advertiser/AR17410177287600472065?region=anywhere
                                              â””â”€â”€â”€â”€â”€â”€â”€â”€ the ID â”€â”€â”€â”€â”€â”€â”˜

For a list of websites: run the Actor in advertisers mode. It returns one row per advertiser account that runs ads pointing to that domain. Here's a real run for hellofresh.com in the US (shortened):

{ "domains": ["hellofresh.com"], "resultType": "advertisers", "regions": ["US"] }

[
  { "advertiserId": "AR00930880903813529601", "name": "Grocery Delivery E-Services USA Inc.", "countryCode": "US", "verified": true, "adsInSample": 186 },
  { "advertiserId": "AR17410177287600472065", "name": "HelloFresh SE", "countryCode": "DE", "verified": true, "adsInSample": 6 },
  { "advertiserId": "AR11301126289508794369", "name": "Launchigo Media FZCO", "countryCode": "AE", "verified": true, "adsInSample": 1 }
]

This is a good example of why this step is worth doing. HelloFresh's US ads run under a legal entity whose name doesn't include "HelloFresh", the German parent company has its own account, and there's a third-party account (an affiliate or agency) sending ads to the same domain. Watching by website catches all three. Watching by advertiser ID gives you exactly one account and nothing else. Both work in the workflow (see Customization below). That run cost $0.007.

Step 2: import the workflow

Copy the workflow JSON at the end of this post and paste it into an empty n8n canvas. n8n turns pasted JSON into nodes.

It contains no credentials or IDs, so there are three things to connect:

Apify token. In Apify Console go to Settings â†’ API & Integrations and copy your API token. In n8n, open Fetch new competitor ads â†’ Credential for Header Auth â†’ Create new: name Authorization, value Bearer <your token>.
Slack. Connect your Slack account in Post new ads to Slack and Post failure notice to Slack, and pick a channel in each. The failure notice can go to a different channel than the alerts if you like.
Your competitors. Open Set competitors and alert settings:

Field	Example	What it does
`competitorDomains`	`hellofresh.com, blueapron.com`	Websites the ads point to, comma-separated
`regions`	`US` or `US, GB`	Countries where the ads were shown. Leave empty for anywhere
`stateKey`	`competitor-ads-daily`	Name of the "already seen" memory. One per workflow
`maxAdsPerCompetitor`	`50`	How many ads to fetch per competitor per run
`includeAdText`	`true`	Headline/description as Google shows it (+$0.30 per 1,000 ads)
`maxCostPerRunUsd`	`0.25`	Hard spending cap per run. Apify's minimum is $0.10

Step 3: how the Apify call works

Fetch new competitor ads is a plain HTTP Request node. It calls Apify's run-sync-get-dataset-items endpoint, which starts the Actor, waits for it to finish and returns the results as JSON in one request:

POST https://api.apify.com/v2/acts/cleandata-labs~google-ads-transparency-explorer/run-sync-get-dataset-items
     ?maxTotalChargeUsd={{ $json.maxCostPerRunUsd }}&timeout=280

The body is built from the settings node:

{{ JSON.stringify({
  domains: String($json.competitorDomains || '').split(',').map(s => s.trim()).filter(Boolean),
  regions: String($json.regions || '').split(',').map(s => s.trim()).filter(Boolean),
  maxAdsPerTarget: $json.maxAdsPerCompetitor,
  includeAdText: $json.includeAdText,
  onlyNew: true,
  stateKey: $json.stateKey
}) }}

Two details I learned the hard way:

Lists must be real JSON arrays. Apify checks the input against the Actor's schema before the run starts, so "domains": "hellofresh.com, blueapron.com" gets rejected with 400 invalid-input: Field input.domains must be array. That's what the .split(',') is for. The same applies to numbers and booleans: "50" and "true" in quotes are rejected too. (Our own README used to claim strings were fine. They're not, and it's fixed now.)
maxTotalChargeUsd can't go below $0.10. Apify refuses the run otherwise. With pay-per-result pricing you're charged for what's delivered, not the cap, so $0.10â€“0.25 is just a safety net.

The node's On Error setting is Continue (using error output). A wrong token, an exhausted credit or a timeout goes to the failure branch and posts a short warning to Slack. Without it, a broken check looks exactly like a quiet day.

Step 4: the first run is your baseline

Click Execute workflow once by hand. The Actor has never seen these competitors, so the first run returns everything that's currently live (up to maxAdsPerCompetitor each), and all of it gets posted. That's your baseline. It's also the most expensive run you'll have (see costs below).

Here's the Slack message from a real baseline run (hellofresh.com, US, 3 ads):

ðŸ“£ 3 new Google ads from the competitors you watch

â€¢ Grocery Delivery E-Services USA Inc. (TEXT, hellofresh.com), first shown 2026-08-06
   View in Transparency Center Â· Creative
â€¢ Grocery Delivery E-Services USA Inc. (IMAGE, hellofresh.com), first shown 2023-06-22
   "HelloFresh Calorie Smart Meal Kit - Two Meals for Two People Per Week | HelloFresh"
   View in Transparency Center
â€¢ Grocery Delivery E-Services USA Inc. (TEXT, hellofresh.com), first shown 2026-03-16
   View in Transparency Center Â· Creative

"New" means new to you: the second ad has been running since 2023, but it's the first time this workflow saw it. From the second run on, only creatives that weren't there before come back.

Two of those text ads have no quoted text. Google archives some text ads as images, so the words are inside the picture behind the Creative link. You aren't charged for the text add-on on those.

Then click Activate. In my tests, the next runs returned 0 new ads and posted nothing, which is exactly what you want. They took 35â€“72 seconds on a large advertiser because the Actor still checks the most recently shown ads (up to 2,000 per competitor) against its memory. That checking is free.

What it costs

The Actor is pay-per-event. The prices below are for Apify's free plan. Paid plans pay less per ad.

What	Price
Each ad delivered	$1.50 per 1,000
Ad text add-on (only when text was found)	+$0.30 per 1,000
Run start	$0.001
Already-seen ads, empty searches, errors	free

What that means for a daily check on two competitors:

Baseline run: at most 2 Ã— 50 ads Ã— $0.0018 = $0.18, usually less. If you want the baseline to fit under the default $0.25 cap with room to spare, set maxAdsPerCompetitor to 30 for the first run.
Quiet day: $0.001. That's about 3 cents a month if nothing changes.
Busy day (a competitor launches 12 new ads): about $0.02.

Apify's free plan includes $5 of credit per month, which covers this many times over. n8n is free self-hosted, or part of your n8n Cloud plan. If you turn includeAdText off, the cost per ad drops by up to 17%, but the alerts become a lot less useful.

What you can't get (and what to do instead)

This is the part most tools gloss over. The Transparency Center is a transparency tool, not a spy tool, and some things simply aren't published.

Spend and budgets. Google publishes spend only for election/political ads. For commercial advertisers nobody has the real number, whatever a tool's landing page says. Workarounds:

Use proxies you can see: how many creatives are active (isActive), how long ads stay live (shownForDays, since an ad that runs for months is probably profitable) and how often new ones appear (this workflow).
For EU-served ads, Google publishes "times shown" ranges per country (the Actor's includeDetails option returns them as impressionsMin/impressionsMax).
Third-party tools (Semrush, SpyFu, Similarweb) estimate search spend from rankings and CPCs. Treat those numbers as estimates, not data.

Targeting outside the EU. Targeting information and reach are only published for ads shown in the EEA, because of the Digital Services Act. For the US, UK and elsewhere you get the creative, its dates and its countries, not the audience. Workarounds:

Use regions as a rough geo-targeting signal: run with ["US"], then ["CA"], and compare. includeDetails also returns per-country first/last shown dates.
If your competitor also advertises in an EU country, includeDetails returns the targeting information Google publishes for those ads. It only describes the EU campaigns, but it's a hint at how they think about audiences.

Keyword search ("who bids on project management software?"). The Transparency Center is organized by advertiser, so you can't search it by keyword, and neither can any tool that reads it. Workarounds:

In your own Google Ads account, Auction Insights lists the domains you compete with in the same auctions.
Search your keywords on Google in the countries you care about and write down the domains in the Sponsored results. Do that for a week and you'll have a solid list.
Feed that list of domains into competitorDomains. From then on, the workflow watches what they launch.

Also good to know: date filters only work for about the last 12 months, landing pages come back as a domain rather than a full URL, and video ads give you a YouTube link rather than a file.

Customization

Watch exact accounts instead of websites: in the HTTP body, replace domains: ... with advertiserIds: ["AR00930880903813529601"]. Or use queries to mix brand names, IDs and Transparency Center links.
Only video or only images: add adFormat: "VIDEO" (or "IMAGE", "TEXT") to the body. Add platforms: ["YOUTUBE"] for YouTube only.
Somewhere other than Slack: swap the two Slack nodes for Gmail, Microsoft Teams, Telegram or Discord. The message is in {{ $json.message }}.
Weekly instead of daily: change the schedule. onlyNew doesn't care how long it's been since the last run.
Start over: change stateKey. The old memory stays in the key-value store google-ads-transparency-state in your Apify account, and you can delete it there.

One rule: don't use the same stateKey in two workflows that might run at the same time. They share one memory, so one will "use up" the new ads before the other sees them.

Wrapping up

There's no Transparency Center API, and BigQuery covers only the EEA and Turkey. But "tell me when a competitor launches something new" is a small, cheap job: a schedule, one HTTP call with onlyNew, and a Slack message. Set it up once, let the baseline run, and from then on you only hear about it (and only pay for it) when something actually changes.

If you try it, I'd like to hear what you're watching and what broke. And if you need something the Transparency Center doesn't publish, the honest answer is usually that no one has it.

The workflow JSON

Click to expand: the full n8n workflow (paste it into an empty canvas)

{
  "name": "Get Slack alerts for new competitor Google ads from the Ads Transparency Center",
  "nodes": [
    {
      "parameters": {
        "content": "## Get a Slack alert when a competitor launches a new Google ad\n\n### How it works\nEvery morning this workflow checks the Google Ads Transparency Center for the competitor websites you list and posts **only the ads they launched since the last check** to Slack: advertiser, format, first-shown date, ad text and links to the ad and its creative.\n\nIt calls the Apify Actor *Google Ads Transparency Scraper* (cleandata-labs) with `onlyNew` on. The Actor remembers which ads it already returned (under your `stateKey`), so repeats are skipped and not charged. The first run returns the current ads as a baseline; after that, quiet days post nothing.\n\n### Setup\n1. Create a free Apify account and copy your API token (Apify Console â†’ Settings â†’ API & Integrations).\n2. In **Fetch new competitor ads**, create a *Header Auth* credential: name `Authorization`, value `Bearer YOUR_APIFY_TOKEN`.\n3. Connect Slack in both Slack nodes and pick your channel.\n4. Edit **Set competitors and alert settings**: websites, countries, `stateKey`.\n5. Run it once by hand to create the baseline, then activate.\n\n### Customization\nSwap Slack for Gmail or Telegram, change the schedule, or set `includeAdText` to false to cut the cost by up to 17%. Cost: about $1.50 per 1,000 new ads on Apify's free plan plus $0.30 per 1,000 for ad text; days with no new ads cost only the run start fee.\n\nDisclosure: the Apify Actor used here is built by the template author.",
        "height": 760,
        "width": 520,
        "color": 1
      },
      "id": "245aea6d-0a32-4ca2-a724-d75a184ac11b",
      "name": "Overview",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        -80,
        -420
      ]
    },
    {
      "parameters": {
        "content": "## 1. Schedule and settings\nRuns daily at 08:00. All values you need to change live in one node.",
        "height": 360,
        "width": 460,
        "color": 7
      },
      "id": "267486d7-7d5c-4324-9575-af7f90c29283",
      "name": "Section config",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        480,
        -120
      ]
    },
    {
      "parameters": {
        "content": "## 2. Fetch only new ads\nOne Apify call per run. `onlyNew` + `stateKey` return just the ads not seen before.",
        "height": 360,
        "width": 460,
        "color": 7
      },
      "id": "3ad1e9bb-37a4-4f9d-863e-97b59d9b7a9f",
      "name": "Section fetch",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        960,
        -120
      ]
    },
    {
      "parameters": {
        "content": "## 3. Post to Slack\nOne message per run, only when there is something new. Failed Apify calls post a notice.",
        "height": 640,
        "width": 700,
        "color": 7
      },
      "id": "1c34be0f-bf09-4bb1-9ee5-a7af6c741d1e",
      "name": "Section alert",
      "type": "n8n-nodes-base.stickyNote",
      "typeVersion": 1,
      "position": [
        1440,
        -120
      ]
    },
    {
      "parameters": {
        "rule": {
          "interval": [
            {
              "triggerAtHour": 8
            }
          ]
        }
      },
      "id": "0e52ac2d-9b2e-4940-9b99-b26bce15ad50",
      "name": "Every day at 08:00",
      "type": "n8n-nodes-base.scheduleTrigger",
      "typeVersion": 1.2,
      "position": [
        540,
        40
      ]
    },
    {
      "parameters": {
        "assignments": {
          "assignments": [
            {
              "id": "f38af90e-43c1-4deb-9e96-1d3c840aaa37",
              "name": "competitorDomains",
              "value": "hellofresh.com, blueapron.com",
              "type": "string"
            },
            {
              "id": "9f004f65-a010-4a41-890d-a26a6ef48d89",
              "name": "regions",
              "value": "US",
              "type": "string"
            },
            {
              "id": "f3ed34fb-04bf-4a4a-8e82-87d784a41ded",
              "name": "stateKey",
              "value": "competitor-ads-daily",
              "type": "string"
            },
            {
              "id": "9a6e281c-de0c-47f6-9aaa-b044b8c0f21f",
              "name": "maxAdsPerCompetitor",
              "value": 50,
              "type": "number"
            },
            {
              "id": "6cd8002d-697a-44bf-8c91-f0b3beb48c51",
              "name": "includeAdText",
              "value": true,
              "type": "boolean"
            },
            {
              "id": "e14e09d0-9ef3-41aa-a5d6-88217613d075",
              "name": "maxCostPerRunUsd",
              "value": 0.25,
              "type": "number"
            }
          ]
        },
        "options": {}
      },
      "id": "2a8a8a02-9f0d-4df0-9e93-cbf665fbc974",
      "name": "Set competitors and alert settings",
      "type": "n8n-nodes-base.set",
      "typeVersion": 3.4,
      "position": [
        740,
        40
      ]
    },
    {
      "parameters": {
        "method": "POST",
        "url": "https://api.apify.com/v2/acts/cleandata-labs~google-ads-transparency-explorer/run-sync-get-dataset-items",
        "authentication": "genericCredentialType",
        "genericAuthType": "httpHeaderAuth",
        "sendQuery": true,
        "queryParameters": {
          "parameters": [
            {
              "name": "maxTotalChargeUsd",
              "value": "={{ $json.maxCostPerRunUsd }}"
            },
            {
              "name": "timeout",
              "value": "280"
            }
          ]
        },
        "sendBody": true,
        "specifyBody": "json",
        "jsonBody": "={{ JSON.stringify({ domains: String($json.competitorDomains || '').split(',').map(s => s.trim()).filter(Boolean), regions: String($json.regions || '').split(',').map(s => s.trim()).filter(Boolean), maxAdsPerTarget: $json.maxAdsPerCompetitor, includeAdText: $json.includeAdText, onlyNew: true, stateKey: $json.stateKey }) }}",
        "options": {
          "timeout": 300000
        }
      },
      "id": "bfe75446-41ee-432b-8e44-6277e88fe17a",
      "name": "Fetch new competitor ads",
      "type": "n8n-nodes-base.httpRequest",
      "typeVersion": 4.2,
      "position": [
        1040,
        40
      ],
      "alwaysOutputData": true,
      "onError": "continueErrorOutput"
    },
    {
      "parameters": {
        "jsCode": "// Turn the Actor rows into one Slack message.\n// Rows: type \"ad\" = a new ad, \"error\" = a target that could not be checked, \"noResults\" = nothing found.\nconst rows = $input.all().map(i => i.json).filter(r => r && r.type);\nconst ads = rows.filter(r => r.type === 'ad');\nconst problems = rows.filter(r => r.type === 'error');\nconst day = s => (s ? String(s).slice(0, 10) : '?');\n// Slack treats < > & as control characters, so escape them in ad text and names.\nconst esc = s => String(s || '').replace(/&/g, '&amp;').replace(/</g, '&lt;').replace(/>/g, '&gt;');\nconst MAX_LINES = 20;\n\nconst lines = ads.slice(0, MAX_LINES).map(ad => {\n  const text = esc((ad.adText || '').replace(/\\s+/g, ' ').trim().slice(0, 140));\n  const media = ad.youtubeUrl || ad.imageUrl || '';\n  return `â€¢ *${esc(ad.advertiserName || ad.domain)}* (${ad.format}, ${ad.domain || ad.searchTarget}), first shown ${day(ad.firstShown)}` +\n    (text ? `\\n   \"${text}\"` : '') +\n    `\\n   <${ad.adUrl}|View in Transparency Center>` + (media ? ` Â· <${media}|Creative>` : '');\n});\n\nlet message = `:mega: *${ads.length} new Google ad${ads.length === 1 ? '' : 's'}* from the competitors you watch`;\nif (lines.length) message += '\\n\\n' + lines.join('\\n');\nif (ads.length > MAX_LINES) message += `\\n\\nâ€¦and ${ads.length - MAX_LINES} more in the Apify dataset.`;\nif (problems.length) {\n  message += `\\n\\n:warning: Could not check: ` + problems.map(p => `${esc(p.target || 'input')} (${esc(p.error)})`).join('; ');\n}\n\nreturn [{ json: { newAdCount: ads.length, problemCount: problems.length, message } }];\n"
      },
      "id": "386e1bba-6e06-4ed5-b479-22099ca9df39",
      "name": "Build Slack summary",
      "type": "n8n-nodes-base.code",
      "typeVersion": 2,
      "position": [
        1500,
        20
      ]
    },
    {
      "parameters": {
        "conditions": {
          "options": {
            "caseSensitive": true,
            "leftValue": "",
            "typeValidation": "strict",
            "version": 2
          },
          "conditions": [
            {
              "id": "3cf489d2-3558-4569-b39f-0ceb37164168",
              "leftValue": "={{ $json.newAdCount + $json.problemCount }}",
              "rightValue": 0,
              "operator": {
                "type": "number",
                "operation": "gt"
              }
            }
          ],
          "combinator": "and"
        },
        "options": {}
      },
      "id": "e666bc7a-76d5-4a63-bfd0-429ea5c2156a",
      "name": "Anything new to report?",
      "type": "n8n-nodes-base.if",
      "typeVersion": 2.2,
      "position": [
        1720,
        20
      ]
    },
    {
      "parameters": {
        "select": "channel",
        "channelId": {
          "__rl": true,
          "value": "",
          "mode": "list",
          "cachedResultName": ""
        },
        "text": "={{ $json.message }}",
        "otherOptions": {}
      },
      "id": "ba01487a-19db-44b2-b2b8-4c35fe2f0a63",
      "name": "Post new ads to Slack",
      "type": "n8n-nodes-base.slack",
      "typeVersion": 2.3,
      "position": [
        1960,
        0
      ]
    },
    {
      "parameters": {
        "select": "channel",
        "channelId": {
          "__rl": true,
          "value": "",
          "mode": "list",
          "cachedResultName": ""
        },
        "text": "=:warning: Competitor ad check failed: {{ $json.error && $json.error.message ? $json.error.message : JSON.stringify($json.error || $json) }}\nCheck your Apify token and the run log in Apify Console.",
        "otherOptions": {}
      },
      "id": "8b0a6862-8206-48f0-ab53-6948760eeedf",
      "name": "Post failure notice to Slack",
      "type": "n8n-nodes-base.slack",
      "typeVersion": 2.3,
      "position": [
        1960,
        360
      ]
    }
  ],
  "connections": {
    "Every day at 08:00": {
      "main": [
        [
          {
            "node": "Set competitors and alert settings",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Set competitors and alert settings": {
      "main": [
        [
          {
            "node": "Fetch new competitor ads",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Fetch new competitor ads": {
      "main": [
        [
          {
            "node": "Build Slack summary",
            "type": "main",
            "index": 0
          }
        ],
        [
          {
            "node": "Post failure notice to Slack",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Build Slack summary": {
      "main": [
        [
          {
            "node": "Anything new to report?",
            "type": "main",
            "index": 0
          }
        ]
      ]
    },
    "Anything new to report?": {
      "main": [
        [
          {
            "node": "Post new ads to Slack",
            "type": "main",
            "index": 0
          }
        ]
      ]
    }
  },
  "pinData": {},
  "settings": {
    "executionOrder": "v1"
  },
  "meta": {
    "templateCredsSetupCompleted": false
  }
}

How to convert web pages and PDFs to clean Markdown for LLMs and RAG (2026)

CleanData Labs — Wed, 28 Oct 2026 15:15:15 +0000

If you feed web pages to an LLM, the first problem isn't the model. It's the input. A typical page is mostly navigation, scripts, cookie banners, footers and class names, and the text you actually care about is a small part of it. You pay for every one of those tokens, and the noise makes retrieval worse too.

This guide shows how to turn web pages and PDFs into clean Markdown that is ready for a prompt or a vector database: first in Apify Console without code, then from Python, then as a tool for AI agents over MCP.

The problem: raw HTML is mostly noise

To put a number on it, I took the Wikipedia article Retrieval-augmented generation and compared three versions of the same page:

Version	Characters	Approx. tokens
Raw HTML (downloaded with curl)	252,873	~63,000
Main content as Markdown, default settings	29,041	7,260
Markdown without references, edit links and hyperlinks	10,752	2,688

The token column is characters divided by 4, which is a rough rule of thumb, not your model's tokenizer. The exact count will differ, but the order of magnitude won't: the raw HTML is roughly ten times bigger than the article text, and most of what you'd send is markup.

Markdown is a good target format because it keeps the structure that matters to a model (headings, lists, tables, links) and drops everything else. Headings also give you natural places to split a document into chunks.

Option 1: write it yourself

For a handful of sites you control, a small script is often enough. trafilatura extracts the main content and can output Markdown directly, readability-lxml is a Python port of the classic Readability algorithm, and markdownify turns any HTML into Markdown. Something like trafilatura.extract(html, output_format="markdown") gets you surprisingly far.

The work starts when you go beyond a few pages: JavaScript-rendered sites that return an empty shell over plain HTTP, PDFs, robots.txt, retries on 429s, relative links, duplicate pages in a crawl, chunking, and somewhere to run it all on a schedule. There are also hosted services for this, such as Apify's own RAG Web Browser, Jina Reader and Firecrawl, and they're worth comparing on price and output for your pages.

Option 2: a hosted URL to Markdown Actor

For the rest of this walkthrough I'll use URL to Markdown Converter, an Actor on the Apify platform (disclosure: built by CleanData Labs). It uses trafilatura and markdownify under the hood, so the extraction is similar to what you'd write yourself, and adds the parts around it:

main-content extraction (article mode), full-page Markdown (full), or auto, which falls back to the full page when too little text is found,
output as markdown, text or cleaned html,
text-based PDFs converted the same way as web pages,
a headless browser only when the plain HTTP result is too thin (renderJs: "auto"),
small crawls by following links or reading the sitemap, with include/exclude URL patterns,
metadata per page (title, author, publish date, language, canonical URL) and approxTokens,
optional chunks for RAG, with overlap.

It respects robots.txt by default and sends an identifiable User-Agent.

Pricing: $0.002 per converted page, and PDFs cost the same as web pages. Failed, blocked, robots.txt-disallowed and duplicate URLs are free. Apify adds its standard $0.00005 Actor start fee per run. The free Apify plan includes $5 of monthly credit, which covers a lot of test pages.

Step-by-step in Apify Console

Open the Actor page and click Try for free. Sign in or create a free Apify account.
Paste your pages or PDF links into URLs. Replace the two prefilled demo URLs with your own.
Leave Output formats on markdown unless you need plain text or HTML as well. Every extra format makes the output bigger.
Leave Extraction mode on Auto, or pick Main content only (article) if you never want menus and footers.
To trim more, use Remove elements (CSS selector), for example .newsletter, .related-posts, #comments. Turn off Keep links if you don't need URLs in the text; it saves tokens.
For RAG, set Chunk size (approx. tokens) to something like 800 and Chunk overlap (approx. tokens) to 80.
Click Start, then open the Output tab. The overview table shows title, language, words and ~tokens per page, and you can export everything as JSON, CSV or Excel.

For a JavaScript-heavy single-page app, set JavaScript rendering to Always and, if content loads late, add a Wait for selector (browser) such as main article.

What the output looks like

This is a shortened row from a real run on the Wikipedia page above, with references and links removed and chunking on:

{
  "url": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "statusCode": 200,
  "title": "Retrieval-augmented generation - Wikipedia",
  "publishedAt": "2023-11-05",
  "siteName": "Wikimedia Foundation, Inc.",
  "language": "en",
  "canonicalUrl": "https://en.wikipedia.org/wiki/Retrieval-augmented_generation",
  "fetchedWith": "http",
  "markdown": "# Retrieval-augmented generation\n\n**Retrieval-augmented generation** (**RAG**) is a technique that enables large language models (LLMs) to retrieve and incorporate new information from external data sources. ...",
  "characters": 10752,
  "words": 1590,
  "approxTokens": 2688,
  "truncated": false,
  "chunks": [
    {"index": 0, "text": "# Retrieval-augmented generation\n\n...", "approxTokens": 722}
  ]
}

fetchedWith tells you how the page was read: http, browser or pdf. Failed URLs come back as rows with an error field (and blocked: true for 401/403/429 and bot walls), so you can see what went wrong without paying for it. That run converted one page and cost $0.00205 ($0.002 for the page plus the start fee). It took about 20 seconds end to end, most of it container startup.

Python example with apify-client

Install the official client with pip install apify-client and copy your API token from Settings → API & Integrations in Apify Console. This is the input from the run above, tested with apify-client 3.4:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run_input = {
    "urls": ["https://en.wikipedia.org/wiki/Retrieval-augmented_generation"],
    "outputFormats": ["markdown"],
    "includeLinks": False,
    "removeElementsCssSelector": ".reflist, .mw-references-wrap, sup.reference, .mw-editsection, .navbox",
    "chunkSizeTokens": 800,
    "chunkOverlapTokens": 80,
}

run = client.actor("cleandata-labs/url-to-markdown").call(
    run_input=run_input,
    max_total_charge_usd=0.10,  # hard cost cap for this run
)

for page in client.dataset(run.default_dataset_id).iterate_items():
    if page.get("error"):
        print("failed:", page["url"], page["error"])
        continue
    print(page["title"], page["approxTokens"], "tokens,", len(page["chunks"]), "chunks")

Output:

Retrieval-augmented generation - Wikipedia 2688 tokens, 4 chunks

Two notes from running it. max_total_charge_usd is a useful safety net for crawls, because the Actor stops cleanly when the cap is reached, but Apify won't accept a value below $0.10. And the CSS selectors in removeElementsCssSelector are site-specific: the ones above are Wikipedia's reference and edit-link classes. For other sites, open the page in your browser's dev tools and look for the wrapper around comments, related posts or newsletter boxes.

Splitting into chunks for RAG

With chunkSizeTokens set, every page gets a chunks list. Chunks are split on paragraph boundaries, and very long paragraphs are cut at the nearest word boundary. In the run above the four chunks came out at 722, 786, 792 and 625 approximate tokens. Each chunk after the first begins with an overlap: up to the last chunkOverlapTokens (here ~80) approximate tokens of the previous chunk, starting at a word boundary. So a chunk can start mid-sentence; that's normal for overlap and usually fine for embeddings.

Here is a minimal way to turn the rows into records for a vector database. embed() stands in for whatever embedding model you use:

records = []
for page in client.dataset(run.default_dataset_id).iterate_items():
    if page.get("error"):
        continue
    for ch in page.get("chunks", []):
        records.append({
            "id": f'{page["finalUrl"]}#{ch["index"]}',
            "text": ch["text"],
            "metadata": {
                "url": page["finalUrl"],
                "title": page["title"],
                "language": page.get("language"),
                "publishedAt": page.get("publishedAt"),
            },
        })

# vectors = embed([r["text"] for r in records])
# then upsert ids, vectors and metadata into Pinecone, Qdrant, pgvector, ...

If you prefer to chunk yourself (by heading, with a real tokenizer, or with LangChain or LlamaIndex splitters), leave chunkSizeTokens at 0 and split the markdown field instead. The Actor's chunker is deliberately simple.

To index a whole docs section rather than single pages, start from the sitemap and limit it:

{
  "urls": ["https://docs.apify.com/"],
  "useSitemap": true,
  "includeUrlGlobs": ["https://docs.apify.com/academy/*"],
  "maxPagesPerStartUrl": 50,
  "chunkSizeTokens": 800
}

You can also follow links with Crawl depth (maxCrawlDepth). Crawls stay on the same host, and duplicate pages (same final URL or same content) are skipped and not charged.

Using it from AI agents via MCP

If you work with Claude, Cursor, VS Code or another MCP client, you can give the agent this Actor as a tool through the official Apify MCP server. The URL below loads only this one Actor:

https://mcp.apify.com?tools=cleandata-labs/url-to-markdown

It's also listed in the official MCP registry as io.github.CleanData-Labs/url-to-markdown.

In Claude Code:

claude mcp add --transport http url-to-markdown "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"

In Cursor (~/.cursor/mcp.json or .cursor/mcp.json in a project):

{
  "mcpServers": {
    "url-to-markdown": {
      "url": "https://mcp.apify.com?tools=cleandata-labs/url-to-markdown"
    }
  }
}

On the first connection the client opens the Apify sign-in (OAuth). If you'd rather use a token, send it as an Authorization: Bearer <YOUR_APIFY_TOKEN> header. Runs are billed to your Apify account at the same $0.002 per page.

The agent then sees a tool called cleandata-labs--url-to-markdown and can be asked things like:

Read https://docs.apify.com/platform/actors as Markdown and summarize it in 5 bullet points.

A tip for agent use: set maxCharacters (for example 20000, around 5k tokens) so a long page doesn't flood the agent's context, and use maxPagesPerStartUrl to keep crawls small. Every row includes approxTokens, so the agent can decide how much of a page to read.

Limitations

Pages behind logins, paywalls or bot protection are not bypassed. They come back as error rows with blocked: true.
PDFs need a text layer. Scanned PDFs aren't OCR'd, and PDF tables come out as plain text.
Crawling is meant for small sections of a site. For crawling an entire large website, use a dedicated crawler.
Token counts are approximate (characters ÷ 4).

Wrapping up

If you only need a few known pages, trafilatura or readability plus markdownify in a short script is a perfectly good answer. If you want PDFs, JavaScript fallback, sitemaps, chunks and an MCP tool without running anything yourself, URL to Markdown Converter is quick to try on the free plan. You can schedule it in Apify Console or trigger it from Make, Zapier or n8n.

Whatever you use, the content still belongs to the sites you fetch it from. Check their terms and copyright before you store, train on or republish it.

How to export Google Trends data to CSV/JSON without an API key (2026)

CleanData Labs — Wed, 28 Oct 2026 12:25:20 +0000

Google Trends is one of the best free signals for demand: is a product category growing, which country searches for it most, and which queries are rising right now. Getting that data out of Google Trends is harder. The website gives you one small CSV per chart, and doing that by hand for 50 keywords is a lost afternoon.

This guide covers why exporting is awkward in 2026, how to export Trends data to CSV, JSON or Excel without code, and how to do the same from Python.

The problem: no open API, and pytrends is archived

There is still no generally available official API. In July 2025 Google announced a Google Trends API alpha. As of October 2026 it is still application-only: the page asks developers to apply as alpha testers, and Google has said it is onboarding a small group. If you have a research use case it is worth applying, but you can't count on access.

pytrends, the popular unofficial library, is no longer maintained. The GeneralMills/pytrends repository is archived, and the last PyPI release (4.9.2) is from April 2023. It can still work, but you are on your own with Google's frequent 429 Too Many Requests responses, changes to the internal endpoints, and proxy rotation.

Manual CSV downloads don't scale. Each chart on trends.google.com has its own download button, and every file has its own header layout.

One thing to remember whatever tool you use: Google Trends values are a relative index from 0 to 100, not search volumes. 100 is the peak popularity within that request (those keywords, that region, that time range). Change the comparison and the numbers change.

Option: a hosted Google Trends scraper

For this walkthrough I'll use Google Trends Scraper, an Actor on the Apify platform (disclosure: built by CleanData Labs). It reads the same public data you see on trends.google.com and returns:

interest over time (with isPartial flags and per-keyword averages),
interest by region (country, state, city or US metro),
related queries and related topics (top and rising, with a numeric growthPercent and an isBreakout flag),
daily Trending Now searches per country.

It goes through Apify Proxy by default and retries rate-limited requests with a new session and a new IP. If the proxy isn't available to your account, it carries on without it instead of failing the run. Anything that still fails ends up in the dataset as an error row with a plain-English reason (and isn't charged), so nothing silently disappears.

Pricing: $0.0025 per result, where one result is a complete data block (for example, a full 12-month timeline or a full region table), not one row per date. Error rows and "no data" rows are free, and Apify adds a $0.00005 Actor start fee per run. Apify's free plan includes $5 of monthly credit, so you can try it without paying. If you want a hard limit, set a maximum cost per run in the run options; the Actor stops cleanly when it gets there.

Step-by-step: export Google Trends data in Apify Console

Open the Actor page and click Try for free. Sign in or create a free Apify account.
In Keywords / search terms, replace the example terms (chatgpt, gemini) with your own, e.g. electric bike and e-scooter. You can also paste a full chart link into Google Trends URLs; the keywords, country, time range, category and property are read from the URL. If you only want the URL, clear the keyword list, otherwise the example keywords run too.
Turn on Compare keywords together if you want the keywords on one shared 0–100 scale (up to 5 per comparison, like on the website). Leave it off to get a separate chart for each keyword. If you compare more than 5, the Actor splits them into groups of 5, and each group gets its own scale (see the anchor-term tip below).
Set Countries / regions (it starts with US; use e.g. GB, or US-CA for California; leave it empty for worldwide). You don't have to be strict about the format: us becomes US and UK becomes GB. Then pick a Time range, such as Past 12 months, or use Custom time range (optional) for exact dates like 2025-01-01 2025-12-31 (2025-01-01 to 2025-12-31 works too). If a code is invalid, you get an error row that says so instead of a silent empty result.
Tick the data you need: Interest over time, Interest by region, Related queries (top + rising), Related topics (top + rising). (Google only returns related topics for single-keyword queries, so leave Compare off if you need them.) For daily trending searches, add country codes to Trending Now countries.
If you plan to open the file in Excel or Google Sheets, turn on Flatten output (CSV / Sheets friendly). Each date or region then becomes its own row instead of a nested array. The price stays the same, because billing counts data blocks, not rows.
Click Start. A small run like this usually finishes in well under a minute. Open the Output tab and export as JSON, CSV, Excel, XML or HTML.

Every interest and related-queries/topics row includes a trendsUrl, so you can open the same chart on trends.google.com and check it. Trending Now rows have an exploreUrl instead.

What the output looks like

Here is a shortened interest-over-time row from a real run (US, past 12 months, the two keywords compared):

{
  "type": "interest_over_time",
  "keywords": ["electric bike", "e-scooter"],
  "geo": "US",
  "timeRange": "today 12-m",
  "timeline": [
    {"date": "Oct 26 – Nov 1, 2025", "timestamp": 1761436800, "isPartial": false, "electric bike": 13, "e-scooter": 0},
    {"date": "Nov 2 – 8, 2025", "timestamp": 1762041600, "isPartial": false, "electric bike": 16, "e-scooter": 1}
  ],
  "averages": {"electric bike": 31, "e-scooter": 1},
  "pointCount": 53
}

And one entry from the "rising" list of a related-queries row from the same run:

{"rank": 2, "query": "yozma", "value": 4100, "formattedValue": "+4,100%", "isBreakout": false, "growthPercent": 4100}

"Breakout" entries (growth over 5,000%) come with isBreakout: true and growthPercent: null, so you can filter them without parsing strings. In this run, "yozma in 10" was the breakout for "electric bike".

Python example with apify-client

To automate this, install the official client (pip install apify-client) and get your API token from Settings → API & Integrations in Apify Console. I tested this with apify-client 3.4:

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run_input = {
    "keywords": ["electric bike", "e-scooter"],
    "compareKeywords": True,
    "geos": ["US"],
    "timeRange": "today 12-m",
    "interestOverTime": True,
    "interestByRegion": True,
    "regionResolution": "REGION",
    "relatedQueries": True,
}

run = client.actor("cleandata-labs/google-trends-explorer").call(run_input=run_input)

for item in client.dataset(run.default_dataset_id).iterate_items():
    if item["type"] == "interest_over_time":
        for point in item["timeline"]:
            print(point["date"], point["electric bike"], point["e-scooter"])

That run returned four results: one timeline, one table of 51 US regions, and one related-queries list per keyword, so it cost about $0.01. To cap spend from code, pass max_total_charge_usd=Decimal("0.50") (from decimal import Decimal) to .call().

To get a CSV file directly, call the dataset items endpoint with format=csv, for example https://api.apify.com/v2/datasets/<DATASET_ID>/items?format=csv. Combine it with flattenOutput: true for clean columns. With the JavaScript client the same call is client.actor("cleandata-labs/google-trends-explorer").call(input).

Tips for comparing keywords

Compare in one request. Values from separate requests are each scaled to their own peak, so "62" in one chart and "62" in another are not comparable. Put the terms in one comparison (compareKeywords: true).
Use an anchor term for more than 5 keywords. Split them into groups of 4 plus one shared reference keyword, then rescale each group by the anchor's values.
Watch the last data point. Points with isPartial: true cover a period that hasn't ended yet and usually look lower than they will end up.
Use topics for ambiguous words. Turn on Keyword / topic suggestions to get Google's topic IDs (e.g. /m/...). Topics group spellings and languages; plain search terms don't.
Look at the scale. In the run above, "e-scooter" averaged 1 next to "electric bike". That doesn't mean nobody searches for it. Next to a much bigger term, small ones get rounded down toward 0.

Tips for regional data

Use ISO codes in Countries / regions: DE for Germany, US-TX for Texas.
Region resolution controls the breakdown: COUNTRY (worldwide), REGION (states/provinces), CITY, or DMA (US metro areas).
Keep Only regions with data on (the default) to drop regions where Google reports nothing.
Region values are relative too: 100 is the region with the highest share of searches for the term, not the most searches.
Apify Proxy is on by default. For bigger runs (more than about 20 queries), switch the proxy group to Residential, because Google rate-limits busy datacenter IPs. If the log shows a lot of retries, raise Delay between requests (s) from the default 1 second.

Using it from an AI agent (MCP)

If you'd rather ask "is interest in e-bikes growing in Germany?" than click through a form, the same Actor works as a tool for Claude, ChatGPT, Cursor, VS Code or any other MCP client, through Apify's MCP server:

https://mcp.apify.com?tools=cleandata-labs/google-trends-explorer

It's also listed in the official MCP Registry as io.github.CleanData-Labs/google-trends-explorer. On first connect your client opens the Apify sign-in (or you can send Authorization: Bearer <APIFY_TOKEN>). In Cursor, for example, put this in .cursor/mcp.json:

{
  "mcpServers": {
    "google-trends": { "url": "https://mcp.apify.com?tools=cleandata-labs/google-trends-explorer" }
  }
}

In Claude Code it's one line: claude mcp add --transport http google-trends "https://mcp.apify.com?tools=cleandata-labs/google-trends-explorer".

Most runs finish in 5–10 seconds, so the agent gets the data within a single tool call, at the same $0.0025 per result. Tip for prompts: say "on one scale" when you compare terms, so the agent sets compareKeywords: true.

Wrapping up

Until Google's official API is widely available, you can choose between doing it by hand, maintaining a scraper yourself, or using a hosted one. If you just need clean JSON or CSV for a list of keywords, a hosted Actor like Google Trends Scraper is quick to set up. You can schedule it in Apify Console, connect it to Google Sheets, Make, Zapier or n8n, or hand it to an AI agent over MCP. If you're feeding web content into an LLM or RAG pipeline next, I wrote a companion guide: How to convert web pages and PDFs to clean Markdown for LLMs and RAG.

Whatever tool you use, the data is the public, aggregated information Google shows on its website. Check Google's terms and the rules that apply to your use.

DEV Community: CleanData Labs

Get an alert when a competitor launches a new Google ad (n8n, no code, pay only for new ads)

The honest answer first: there's no official API

What we'll build

Step 1: find your competitor's advertiser (the AR... ID)

Step 2: import the workflow

Step 3: how the Apify call works

Step 4: the first run is your baseline

What it costs

What you can't get (and what to do instead)

Customization

Wrapping up

The workflow JSON

How to convert web pages and PDFs to clean Markdown for LLMs and RAG (2026)

The problem: raw HTML is mostly noise

Option 1: write it yourself

Option 2: a hosted URL to Markdown Actor

Step-by-step in Apify Console

What the output looks like

Python example with apify-client

Splitting into chunks for RAG

Using it from AI agents via MCP

Limitations

Wrapping up

How to export Google Trends data to CSV/JSON without an API key (2026)

The problem: no open API, and pytrends is archived

Option: a hosted Google Trends scraper

Step-by-step: export Google Trends data in Apify Console

What the output looks like

Python example with apify-client

Tips for comparing keywords

Tips for regional data

Using it from an AI agent (MCP)

Wrapping up

Step 1: find your competitor's advertiser (the `AR...` ID)