<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Fabian</title>
    <description>The latest articles on DEV Community by Fabian (@fade97).</description>
    <link>https://dev.to/fade97</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4100671%2Fc8d8a269-b56c-47aa-901c-ec8400f56796.png</url>
      <title>DEV Community: Fabian</title>
      <link>https://dev.to/fade97</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/fade97"/>
    <language>en</language>
    <item>
      <title>Apify limits I only found out about by hitting them</title>
      <dc:creator>Fabian</dc:creator>
      <pubDate>Sat, 21 Nov 2026 08:04:48 +0000</pubDate>
      <link>https://dev.to/fade97/apify-limits-i-only-found-out-about-by-hitting-them-34he</link>
      <guid>https://dev.to/fade97/apify-limits-i-only-found-out-about-by-hitting-them-34he</guid>
      <description>&lt;p&gt;I've been publishing scrapers on the Apify Store since August 2026, 24 public actors at this point. Most of the platform is well documented, but a handful of limits I only learned about when a request came back with an error code I hadn't seen before. Here they are, with the error you get, so the next person can search for it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Publishing and the Store
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;5 actor publications per day.&lt;/strong&gt; Making an actor public (or re-publishing it) counts against a daily budget of 5. The sixth one returns &lt;code&gt;daily-publication-limit-exceeded&lt;/code&gt;. Hand-retrying in a loop doesn't help, it resets daily. I ended up with a small timer script that retries the pending ones once in a while.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Example tasks: 50 per actor in the docs, 10 published per actor in practice.&lt;/strong&gt; Published tasks get their own landing page under &lt;code&gt;apify.com/&amp;lt;user&amp;gt;/&amp;lt;actor&amp;gt;/examples/&amp;lt;task&amp;gt;&lt;/code&gt;, which is great for search. The docs say 50 per actor. What I ran into was &lt;code&gt;too-many-public-actor-tasks-per-actor&lt;/code&gt; at 10, plus an account-wide cap of 100 published tasks. So with more than 10 actors you have to decide which ones get pages.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Publishing needs an output schema now.&lt;/strong&gt; &lt;code&gt;PUT /v2/acts/{id}&lt;/code&gt; with &lt;code&gt;isPublic: true&lt;/code&gt; fails with &lt;code&gt;schemas-required&lt;/code&gt; if the default build has no &lt;code&gt;output&lt;/code&gt; section in &lt;code&gt;.actor/actor.json&lt;/code&gt;. Adding it means a rebuild first, then the publish works.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Icons can't be set through the API.&lt;/strong&gt; &lt;code&gt;pictureUrl&lt;/code&gt; only accepts Apify's own uploads, so icons are a manual step in the Console. Reading the field back through &lt;code&gt;GET /v2/acts/{id}&lt;/code&gt; at least tells you which actors still have none.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;&lt;code&gt;defaultRunOptions&lt;/code&gt; in &lt;code&gt;actor.json&lt;/code&gt; was not applied by &lt;code&gt;apify push&lt;/code&gt;.&lt;/strong&gt; I set default memory in the file and the actor kept the old value. Setting it through the API (&lt;code&gt;PUT /v2/acts/{id}&lt;/code&gt; with &lt;code&gt;defaultRunOptions&lt;/code&gt;) worked.&lt;/p&gt;

&lt;h2&gt;
  
  
  Pricing
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Price increases start 14 days in the future.&lt;/strong&gt; Pay-per-event prices are a list of &lt;code&gt;pricingInfos&lt;/code&gt; entries. A new entry that raises a price has to start at least 14 days out, and Apify then emails existing users. Decreases can take effect immediately.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;One price change per actor per month.&lt;/strong&gt; Any new &lt;code&gt;pricingInfos&lt;/code&gt; entry counts, including a decrease or a fix to an increase that hasn't started yet. The second one in a month returns HTTP 403 &lt;code&gt;cannot-modify-actor-pricing-too-frequently&lt;/code&gt;, with the next allowed date. A scheduled entry also can't be edited or deleted: the PUT has to contain the existing entries unchanged plus at most one new one. So measure first, then set the price.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The start event is charged per GB.&lt;/strong&gt; &lt;code&gt;apify-actor-start&lt;/code&gt; is charged automatically by the platform (calling &lt;code&gt;Actor.charge&lt;/code&gt; on it fails). It's charged once for the first GB of memory and once more for every additional GB. A $0.01 start price on an actor with 4 GB default memory is $0.04 per run. My HTTP scrapers peak at 50 to 130 MB, so 1024 MB is now the default everywhere.&lt;/p&gt;

&lt;h2&gt;
  
  
  Running things
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;Concurrent runs: HTTP 402, and no run at all.&lt;/strong&gt; Testing my own paid actors, starting about 5 runs at once returned HTTP 402 on run creation. No run is created, so nothing shows up as failed anywhere. I test in batches of 3 now.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The plan's usage limit is a hard stop.&lt;/strong&gt; On the free plan the $5 monthly credit is a cap: when it's used up, runs and API calls are blocked until the usage cycle resets. The account also dropped back to the free plan once without me noticing, so before a test batch I read &lt;code&gt;GET /v2/users/me/limits&lt;/code&gt; instead of assuming the plan.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Maximum cost per run ends as ABORTED.&lt;/strong&gt; When a pay-per-event run reaches the user's max cost, the platform aborts it, often before the actor can finish cleanly, and an aborted run counts as a failure. The fix is to cap your own pushes with the SDK's &lt;code&gt;calculateMaxEventChargeCountWithinLimit&lt;/code&gt;, keep one event price in reserve, and exit as SUCCEEDED with a status message that says the budget was reached. The minimum &lt;code&gt;maxTotalChargeUsd&lt;/code&gt; a user can start with is $0.05.&lt;/p&gt;

&lt;h2&gt;
  
  
  Debugging other people's runs
&lt;/h2&gt;

&lt;p&gt;&lt;strong&gt;You can't see them.&lt;/strong&gt; As the owner of a public actor you get no run object, no log and no input for runs started by other users. &lt;code&gt;GET /v2/acts/{id}/runs&lt;/code&gt; returns only your own. What you do get is &lt;code&gt;stats.publicActorRunStats30Days&lt;/code&gt;, a 30-day count of runs per status. I tried to get around this with webhooks; that turned out to have the same limit, which is its own post.&lt;/p&gt;

&lt;p&gt;The practical consequence: the status message is the only thing the user sees when a run fails, and the 30-day failed count is the only thing you see. Both are worth getting right.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Required fields with a &lt;code&gt;default&lt;/code&gt; let empty runs start.&lt;/strong&gt; The platform validates input against the schema before starting the container. A missing required field returns HTTP 400 and no run exists. But a required field that also has &lt;code&gt;"default": []&lt;/code&gt; passes that check, the container starts, your code throws, and the run counts as FAILED in public stats. Required fields get a &lt;code&gt;prefill&lt;/code&gt; and &lt;code&gt;minItems: 1&lt;/code&gt;, never a &lt;code&gt;default&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;None of this is hidden, exactly. It's spread across docs, error messages and the API reference. If you're about to publish your first actor, the 402 on concurrent runs and the one-price-change-per-month rule are the two I'd want to know on day one.&lt;/p&gt;

</description>
      <category>apify</category>
      <category>webscraping</category>
      <category>javascript</category>
      <category>devops</category>
    </item>
    <item>
      <title>YouTube transcripts for RAG without the Data API (and without maintaining a scraper)</title>
      <dc:creator>Fabian</dc:creator>
      <pubDate>Tue, 17 Nov 2026 08:00:24 +0000</pubDate>
      <link>https://dev.to/fade97/youtube-transcripts-for-rag-without-the-data-api-and-without-maintaining-a-scraper-9oe</link>
      <guid>https://dev.to/fade97/youtube-transcripts-for-rag-without-the-data-api-and-without-maintaining-a-scraper-9oe</guid>
      <description>&lt;p&gt;I needed transcripts for a few hundred videos from a couple of channels for a RAG project. The YouTube Data API doesn't give you captions for videos you don't own, and the usual Python transcript library broke on me twice in one month, so I ended up with the setup below. Posting it because the question comes up a lot.&lt;/p&gt;

&lt;p&gt;Two steps: list the videos of a channel, then fetch transcripts for that list. I'm using two Apify actors I wrote for this (they read the public pages and the innertube endpoints, no API key, no quota). The client library is &lt;code&gt;pip install apify-client&lt;/code&gt;.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;from&lt;/span&gt; &lt;span class="n"&gt;apify_client&lt;/span&gt; &lt;span class="k"&gt;import&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;

&lt;span class="n"&gt;client&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;ApifyClient&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"YOUR_APIFY_TOKEN"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 1. all videos of a channel&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pulsedata/youtube-channel-videos-scraper"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"channelUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"https://www.youtube.com/@3blue1brown"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"channelTabs"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"videos"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"maxVideosPerChannel"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;   &lt;span class="c1"&gt;# 0 = no limit&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;videos&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;i&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"type"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="s2"&gt;"video"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="k"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;videos&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"videos"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;

&lt;span class="c1"&gt;# 2. transcripts&lt;/span&gt;
&lt;span class="n"&gt;run&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;actor&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"pulsedata/youtube-transcript-scraper"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;call&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run_input&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="s2"&gt;"videoUrls"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;v&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;v&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;videos&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"languages"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"en"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"fallbackToAnyLanguage"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"outputFormats"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"segments"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="s2"&gt;"includeMetadata"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="kc"&gt;True&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;docs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;client&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;dataset&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;run&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"defaultDatasetId"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;iterate_items&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="ow"&gt;not&lt;/span&gt; &lt;span class="n"&gt;d&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;get&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s2"&gt;"error"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Each item in &lt;code&gt;docs&lt;/code&gt; has &lt;code&gt;text&lt;/code&gt; (whole transcript), &lt;code&gt;segments&lt;/code&gt; (list of &lt;code&gt;{start, duration, text}&lt;/code&gt;) and metadata: &lt;code&gt;title&lt;/code&gt;, &lt;code&gt;channelName&lt;/code&gt;, &lt;code&gt;durationSeconds&lt;/code&gt;, &lt;code&gt;publishDate&lt;/code&gt;, &lt;code&gt;url&lt;/code&gt;. Items with an &lt;code&gt;error&lt;/code&gt; field are videos without captions, I just drop those.&lt;/p&gt;

&lt;h2&gt;
  
  
  Chunking
&lt;/h2&gt;

&lt;p&gt;I first chunked by a fixed number of tokens and it was fine, but chunking along the transcript segments works a bit better because the segment boundaries follow the speaker's pauses, and you get a start time for free. This groups segments until roughly 300 words:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="k"&gt;def&lt;/span&gt; &lt;span class="nf"&gt;chunks&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;max_words&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;300&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
    &lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt; &lt;span class="k"&gt;in&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"segments"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="k"&gt;is&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="n"&gt;start&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
        &lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;append&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="nb"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;s&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
        &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;&amp;gt;=&lt;/span&gt; &lt;span class="n"&gt;max_words&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
            &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
                   &lt;span class="s2"&gt;"channel"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"channelName"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{doc["url"]}&amp;amp;t={int(start)}'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
            &lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;n&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="kc"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="mi"&gt;0&lt;/span&gt;
    &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt;
        &lt;span class="k"&gt;yield&lt;/span&gt; &lt;span class="p"&gt;{&lt;/span&gt;&lt;span class="s2"&gt;"text"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="s2"&gt;" "&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;join&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;buf&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"start"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;start&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"title"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
               &lt;span class="s2"&gt;"channel"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;doc&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="s2"&gt;"channelName"&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="s2"&gt;"url"&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="n"&gt;f&lt;/span&gt;&lt;span class="s1"&gt;'{doc["url"]}&amp;amp;t={int(start)}'&lt;/span&gt;&lt;span class="p"&gt;}&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;The &lt;code&gt;&amp;amp;t=&lt;/code&gt; on the URL is the useful part. When the model answers it can link to the exact spot in the video, which people like a lot more than a link to the video.&lt;/p&gt;

&lt;h2&gt;
  
  
  Cost, and things that bit me
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Listing is $1 per 1,000 videos, transcripts $3 per 1,000. A 500 video channel is about $2 total. Apify's free tier ($5/month) covers the first few channels.&lt;/li&gt;
&lt;li&gt;Auto-generated captions are fine for retrieval. For quoting them verbatim they're not great, punctuation is missing and names get mangled. The &lt;code&gt;isAutoGenerated&lt;/code&gt; flag tells you which is which.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;translateTo: "en"&lt;/code&gt; gives you English text for non-English channels (YouTube's own translation, quality varies).&lt;/li&gt;
&lt;li&gt;If you already have the video URLs, skip step 1. The transcript actor also takes playlist URLs directly.&lt;/li&gt;
&lt;li&gt;Shorts have transcripts too but they're usually a few sentences, I filter them out with &lt;code&gt;isShort&lt;/code&gt; from step 1.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Actors: &lt;a href="https://apify.com/pulsedata/youtube-channel-videos-scraper" rel="noopener noreferrer"&gt;https://apify.com/pulsedata/youtube-channel-videos-scraper&lt;/a&gt; and &lt;a href="https://apify.com/pulsedata/youtube-transcript-scraper" rel="noopener noreferrer"&gt;https://apify.com/pulsedata/youtube-transcript-scraper&lt;/a&gt;. If one of them fails on a particular channel, tell me which one, that's the quickest way for me to find out.&lt;/p&gt;

</description>
      <category>python</category>
      <category>ai</category>
      <category>rag</category>
      <category>youtube</category>
    </item>
    <item>
      <title>I scraped the US top charts (840 apps). Some numbers on age and monetization</title>
      <dc:creator>Fabian</dc:creator>
      <pubDate>Sun, 15 Nov 2026 07:48:47 +0000</pubDate>
      <link>https://dev.to/fade97/i-scraped-the-us-top-charts-840-apps-some-numbers-on-age-and-monetization-263o</link>
      <guid>https://dev.to/fade97/i-scraped-the-us-top-charts-840-apps-some-numbers-on-age-and-monetization-263o</guid>
      <description>&lt;p&gt;I've been going back and forth on whether to build an Android app as a side project, and instead of reading more opinion pieces I scraped the top charts of the App Store and Google Play (US, main categories, 840 apps total, 426 Play and 414 App Store) and looked at what's actually in there.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Age.&lt;/strong&gt; Google Play doesn't show release dates but the App Store does, so for that half: the median app is 9.6 years old. 47% are older than 10 years. 13% are under 2 years. The charts are basically a list of things that have been around since 2016 or earlier. I sort of knew this, but seeing 47% over ten years was still a bit deflating.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Price.&lt;/strong&gt; All 840 are free to download. Not most, all. I looked for paid apps in Productivity and Education specifically because that's where they used to be. Zero.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Monetization&lt;/strong&gt;, Google Play half only (the App Store doesn't expose the flags): 62% have in-app purchases, 38% show ads. By category it's quite different:&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Category&lt;/th&gt;
&lt;th&gt;ads&lt;/th&gt;
&lt;th&gt;IAP&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;Entertainment&lt;/td&gt;
&lt;td&gt;46%&lt;/td&gt;
&lt;td&gt;51%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Productivity&lt;/td&gt;
&lt;td&gt;27%&lt;/td&gt;
&lt;td&gt;23%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Health &amp;amp; Fitness&lt;/td&gt;
&lt;td&gt;17%&lt;/td&gt;
&lt;td&gt;39%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Finance&lt;/td&gt;
&lt;td&gt;10%&lt;/td&gt;
&lt;td&gt;25%&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;Education&lt;/td&gt;
&lt;td&gt;9%&lt;/td&gt;
&lt;td&gt;27%&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;So ads are only really common in Entertainment. Health &amp;amp; Fitness is subscriptions. Finance apps mostly make money outside the app, which makes sense (banks, brokers, the app is a front end).&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Publishers.&lt;/strong&gt; 647 different ones for 840 apps. The top 10 (Microsoft and Google with 18 each, then Disney with 7) have under 10% of the slots between them. I expected a lot more concentration than that.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Updates.&lt;/strong&gt; 90% were updated in the last 30 days, 96% in the last 90. Median rating 4.65.&lt;/p&gt;

&lt;p&gt;What I take from it for my own project: don't plan on the charts, plan on a niche with search traffic. Free download with a one-time unlock or a subscription depending on the category. Monthly updates are the baseline, not a nice to have. And ad-supported is only a normal model in Entertainment, everywhere else you'd be the odd one out.&lt;/p&gt;

&lt;p&gt;Usual caveats: one snapshot, US only, top charts only (so survivorship bias by construction), and the ads/IAP flags are Play only.&lt;/p&gt;

&lt;p&gt;The scraper is on Apify (&lt;a href="https://apify.com/pulsedata/app-store-google-play-scraper" rel="noopener noreferrer"&gt;https://apify.com/pulsedata/app-store-google-play-scraper&lt;/a&gt;) if you want to run it for another country or category, it does both stores in one run. The dataset with all fields (rating histogram, installs, IAP price ranges, update dates and so on) is $9 on Gumroad: &lt;a href="https://pulsdata.gumroad.com/l/app-market-us-2026/LAUNCH" rel="noopener noreferrer"&gt;https://pulsdata.gumroad.com/l/app-market-us-2026/LAUNCH&lt;/a&gt;&lt;/p&gt;

</description>
      <category>mobile</category>
      <category>android</category>
      <category>ios</category>
      <category>data</category>
    </item>
    <item>
      <title>Google ads from 300 DTC brands: most of them run way longer than I thought</title>
      <dc:creator>Fabian</dc:creator>
      <pubDate>Sat, 29 Aug 2026 19:11:50 +0000</pubDate>
      <link>https://dev.to/fade97/google-ads-from-300-dtc-brands-most-of-them-run-way-longer-than-i-thought-386i</link>
      <guid>https://dev.to/fade97/google-ads-from-300-dtc-brands-most-of-them-run-way-longer-than-i-thought-386i</guid>
      <description>&lt;p&gt;I built a scraper for the Google Ads Transparency Center a couple of weeks ago (that's the site where Google lists every ad an advertiser runs, with first and last shown dates). To test it properly I needed a decent sized list, so I took around 300 DTC / ecommerce brands, ran it, and ended up with 3,994 ads. Then I got curious and started looking at the dates instead of just checking whether the scraper worked.&lt;/p&gt;

&lt;p&gt;The number I keep coming back to is the median: 227 days. That's per ad, not per brand. So the typical ad in this set has been shown for over seven months. 36% are older than a year, 16% older than two years. Only 11% are under a month old.&lt;/p&gt;

&lt;p&gt;That is not what I would have guessed. Everything I read about DTC paid media is about creative testing, rotating weekly, killing losers fast etc. Maybe that's true on Meta and it just doesn't show up in Google's data, or maybe the testing happens in a small share of ads while the bulk keeps running. I don't know. But the "bulk keeps running" part is definitely in the data.&lt;/p&gt;

&lt;p&gt;Per brand, median days shown (only brands where I have 30+ ads):&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Adidas 1319&lt;/li&gt;
&lt;li&gt;Bombas 1219&lt;/li&gt;
&lt;li&gt;Booking.com 988&lt;/li&gt;
&lt;li&gt;Warby Parker 940&lt;/li&gt;
&lt;li&gt;Canva 888&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;and the short end:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Calm 28&lt;/li&gt;
&lt;li&gt;AG1 54&lt;/li&gt;
&lt;li&gt;Babbel 55&lt;/li&gt;
&lt;li&gt;GoPro 55&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;Adidas at 1319 days is 3.6 years. Some of those are probably brand search ads nobody has touched since 2022, and that's fine, they work.&lt;/p&gt;

&lt;p&gt;Format: 74% text, 15% image, 11% video. I had assumed video would be bigger. It's mostly Search text ads.&lt;/p&gt;

&lt;p&gt;Some caveats, because there are a few. Google caps how many ads it lists per advertiser, so for the big brands I don't have everything. "Still shown" says nothing about spend, an ad can be live with a tiny budget. "First shown" is when Google first saw it, so for really old ads the number is truncated at whenever the transparency center started tracking. And the brand list is just brands I picked, so US-centric consumer stuff, nothing scientific about the sample.&lt;/p&gt;

&lt;p&gt;Still, if you do competitor research the practical bit is easy: sort by days shown, descending. Whatever a competitor has been running for two years is what works for them. Their newest ad is a test.&lt;/p&gt;

&lt;p&gt;You can look up single advertisers on adstransparency.google.com by hand, the annoying part is it's paginated and you can't export anything. My scraper is on Apify if you want to do it for a list: &lt;a href="https://apify.com/pulsedata/google-ads-transparency-scraper" rel="noopener noreferrer"&gt;https://apify.com/pulsedata/google-ads-transparency-scraper&lt;/a&gt;. It calls the same JSON-RPC endpoints the page uses, nothing clever, the hard part was figuring out the field numbers.&lt;/p&gt;

&lt;p&gt;I put the full dataset (CSV + JSON, all 3,994 ads with dates, format, regions, preview URLs) and a short PDF with charts on Gumroad for $9: &lt;a href="https://pulsdata.gumroad.com/l/google-ads-dtc-2026/LAUNCH" rel="noopener noreferrer"&gt;https://pulsdata.gumroad.com/l/google-ads-dtc-2026/LAUNCH&lt;/a&gt;. If you just want the headline numbers they're all in this post.&lt;/p&gt;

</description>
      <category>marketing</category>
      <category>data</category>
      <category>webscraping</category>
      <category>ecommerce</category>
    </item>
  </channel>
</rss>
