<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Emirhan Güven</title>
    <description>The latest articles on DEV Community by Emirhan Güven (@azorkai).</description>
    <link>https://dev.to/azorkai</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4104121%2F3f1ecced-ea9a-452b-9399-17e5ccffea36.jpg</url>
      <title>DEV Community: Emirhan Güven</title>
      <link>https://dev.to/azorkai</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/azorkai"/>
    <language>en</language>
    <item>
      <title>How we measured Telegram channel reach across 3.2M public channels (and why the median, not the mean)</title>
      <dc:creator>Emirhan Güven</dc:creator>
      <pubDate>Sat, 19 Sep 2026 01:04:06 +0000</pubDate>
      <link>https://dev.to/azorkai/how-we-measured-telegram-channel-reach-across-32m-public-channels-and-why-the-median-not-the-2878</link>
      <guid>https://dev.to/azorkai/how-we-measured-telegram-channel-reach-across-32m-public-channels-and-why-the-median-not-the-2878</guid>
      <description>&lt;p&gt;Telegram prints a view counter on every public channel post. Put that next to the subscriber count and you get a number no other large platform hands you for free: what share of an audience actually sees a post. We computed it across a catalogue of 3,205,346 public channels and published the result as open data. This note is about how the measurement was built, where it is weak, and why the headline figure is a median.&lt;/p&gt;

&lt;p&gt;The short version of the result: on channels with 1,000 to 5,000 subscribers the median post is seen by 31.4% of subscribers. Above 500,000 subscribers it is 3.3%.&lt;/p&gt;

&lt;h2&gt;
  
  
  The definition
&lt;/h2&gt;

&lt;p&gt;For each channel:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;reach = average views per post / subscribers
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Then, for each size band, the median of that ratio across the channels in the band. That is the whole formula. A channel with 10,000 subscribers whose posts average 1,500 views has a reach of 15%.&lt;/p&gt;

&lt;p&gt;Two implementation details matter more than the formula itself.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;The ratio is computed from the two stored inputs, not from a precomputed engagement column.&lt;/strong&gt; The crawler has its own engagement field, but it is still being backfilled, and reading it would make the score exist for some channels and not others with no visible reason. Dividing two columns that are always present is duller and always answers.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Channels under 1,000 subscribers are excluded.&lt;/strong&gt; Below that floor a single forwarded post can put the ratio anywhere, so it is too noisy to rank. The API reports these as &lt;code&gt;below_floor&lt;/code&gt; rather than scoring them zero. The same goes for groups (&lt;code&gt;not_a_channel&lt;/code&gt;: Telegram publishes no per-post views for groups) and for channels the crawler has never seen a view count for (&lt;code&gt;no_view_sample&lt;/code&gt;). None of those is an error, and none of them is silently folded into the statistics.&lt;/p&gt;

&lt;h2&gt;
  
  
  Why the median
&lt;/h2&gt;

&lt;p&gt;The obvious summary is the mean. We did not publish it, for two reasons that pull in opposite directions.&lt;/p&gt;

&lt;p&gt;The first is the tail above 1.0. A ratio above 1.0, more views per post than subscribers, is common on Telegram and is not proof of a real audience. Forwards, cross-posts and paid promotion all deliver views from people who are not on the subscriber list. In our scoring those channels get the verdict &lt;code&gt;amplified&lt;/code&gt;, which says out loud that the reach is arriving from outside. A few channels several times above 1.0 move a band mean a long way and tell you nothing about the typical channel in that band. The median does not move.&lt;/p&gt;

&lt;p&gt;The second is the tail at the bottom. Subscriber counts padded with bought or long-dead accounts push the ratio down. A band mean would be dragged by those as well, in the other direction, and the two distortions do not cancel in any way you can rely on.&lt;/p&gt;

&lt;p&gt;The median is also what a benchmark needs. "Half of the channels this size do better than 6.0%" is a sentence you can act on: a 200,000-subscriber channel reaching 12% is doing twice as well as its peers, and one at 2% deserves a closer look before you pay for a post. A mean gives you no such sentence.&lt;/p&gt;

&lt;p&gt;Internally the median is one point of a 101-point percentile ladder per band. The public page shows the medians; the API exposes the ladders with their sample sizes and rebuild timestamps, so a percentile is never a number with an invisible population behind it.&lt;/p&gt;

&lt;h2&gt;
  
  
  Count everything where you can, sample where you cannot
&lt;/h2&gt;

&lt;p&gt;Five bands, filled two different ways.&lt;/p&gt;

&lt;div class="table-wrapper-paragraph"&gt;&lt;table&gt;
&lt;thead&gt;
&lt;tr&gt;
&lt;th&gt;Band&lt;/th&gt;
&lt;th&gt;Median reach&lt;/th&gt;
&lt;th&gt;Channels measured&lt;/th&gt;
&lt;th&gt;Basis&lt;/th&gt;
&lt;/tr&gt;
&lt;/thead&gt;
&lt;tbody&gt;
&lt;tr&gt;
&lt;td&gt;1k-5k&lt;/td&gt;
&lt;td&gt;31.4%&lt;/td&gt;
&lt;td&gt;7,179&lt;/td&gt;
&lt;td&gt;sample&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;5k-20k&lt;/td&gt;
&lt;td&gt;15.8%&lt;/td&gt;
&lt;td&gt;2,750&lt;/td&gt;
&lt;td&gt;sample&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;20k-100k&lt;/td&gt;
&lt;td&gt;9.1%&lt;/td&gt;
&lt;td&gt;1,201&lt;/td&gt;
&lt;td&gt;sample&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;100k-500k&lt;/td&gt;
&lt;td&gt;6.0%&lt;/td&gt;
&lt;td&gt;8,380&lt;/td&gt;
&lt;td&gt;every channel&lt;/td&gt;
&lt;/tr&gt;
&lt;tr&gt;
&lt;td&gt;500k+&lt;/td&gt;
&lt;td&gt;3.3%&lt;/td&gt;
&lt;td&gt;1,211&lt;/td&gt;
&lt;td&gt;every channel&lt;/td&gt;
&lt;/tr&gt;
&lt;/tbody&gt;
&lt;/table&gt;&lt;/div&gt;

&lt;p&gt;The two large bands are exact because they are small: 8,380 and 1,211 channels is nothing to compute over. The small bands are enormous (the 1k to 10k range alone projects to roughly 338,000 channels), so their ladders are built from a &lt;code&gt;TABLESAMPLE&lt;/code&gt; of the band. The shape is sound; a percentile from a sampled band is an estimate, and the &lt;code&gt;exact: false&lt;/code&gt; flag on each band says so. In the CSV the same flag is the &lt;code&gt;basis&lt;/code&gt; column.&lt;/p&gt;

&lt;p&gt;One consequence worth knowing. The size distribution comes from a separate sample of 14,431 channels with a known subscriber count, and it projects roughly 7,900 channels above 100,000 subscribers. The exact reach tables hold 9,591 (8,380 plus 1,211). The projection rests on a few dozen sampled rows in a band that is about a quarter of one percent of the network; a miss of about 20% is what sampling error looks like at that rarity. When an exact count and a projection disagree, trust the exact count.&lt;/p&gt;

&lt;h2&gt;
  
  
  What we left out, and why
&lt;/h2&gt;

&lt;p&gt;Three catalogue fields did not make the report.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Category.&lt;/strong&gt; Most channels carry no category label at all. Of 15,240 sampled channels, 719 had one. Those counts describe who bothers to label a channel, not Telegram. They are in the JSON with that caveat attached; they are not on the page.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Country.&lt;/strong&gt; The country facet came back empty on the same sampling pass: not one sampled channel carried a value. There was nothing to publish.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;Languages beyond the top four.&lt;/strong&gt; Language is detected automatically from short channel descriptions. For Russian (27.0% of sampled channels), English (12.1%), Arabic (6.0%) and Persian (5.6%) the detector is reliable. Below that it starts confusing related languages, and the long tail ranks some languages in places nobody who has used Telegram would put them. So the page shows four, and the JSON keeps the full tail with its raw sample counts so you can judge the noise yourself.&lt;/p&gt;

&lt;p&gt;The rule we ended up with: a populated field is a claim, not a measurement. Publish the ones you can defend, ship the rest as raw rows.&lt;/p&gt;

&lt;h2&gt;
  
  
  The limits that stay
&lt;/h2&gt;

&lt;p&gt;Telegram's view counter counts every view of a post, including views through forwards and web previews by people who are not subscribers. Reach here can therefore overstate the share of subscribers who saw a post, most of all on channels whose posts travel. Subscriber counts inflated by inactive or bought accounts pull the other way. Bands below 100,000 subscribers are sample estimates. Everything under 1,000 subscribers is outside the reach figures, and by count that is 85.7% of the catalogue.&lt;/p&gt;

&lt;h2&gt;
  
  
  One script, three outputs
&lt;/h2&gt;

&lt;p&gt;The last decision was the boring one. The two API responses (&lt;code&gt;/insights/telegram/reach-bands&lt;/code&gt; and &lt;code&gt;/directory/telegram/facets&lt;/code&gt;, captured on 11 September 2026) sit in the repo as snapshots, and one Python script turns them into the TypeScript constants the page renders, the CSV and the JSON. The page cannot drift from the downloads because none of the numbers is typed by hand.&lt;/p&gt;

&lt;h2&gt;
  
  
  The data
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;Report and tables: &lt;a href="https://pinlyx.com/research/telegram-channel-reach-2026" rel="noopener noreferrer"&gt;https://pinlyx.com/research/telegram-channel-reach-2026&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CSV: &lt;a href="https://pinlyx.com/data/telegram-channel-reach-2026.csv" rel="noopener noreferrer"&gt;https://pinlyx.com/data/telegram-channel-reach-2026.csv&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;JSON, including the category rows and the full language tail: &lt;a href="https://pinlyx.com/data/telegram-channel-reach-2026.json" rel="noopener noreferrer"&gt;https://pinlyx.com/data/telegram-channel-reach-2026.json&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;CC BY 4.0. Use it, quote it, republish it; the only ask is attribution.&lt;/p&gt;

&lt;p&gt;Disclosure: I work on Pinlyx, the CRM whose catalogue these figures come from.&lt;/p&gt;

&lt;p&gt;If you have measured the same ratio on another platform that exposes per-post views, I would like to know whether the tenfold fall from small channels to large ones holds there too.&lt;/p&gt;

</description>
      <category>telegram</category>
      <category>datascience</category>
      <category>opendata</category>
      <category>analytics</category>
    </item>
    <item>
      <title>Google crawled 230 pages of my site and indexed 2. Here is how I diagnosed it.</title>
      <dc:creator>Emirhan Güven</dc:creator>
      <pubDate>Fri, 04 Sep 2026 15:52:13 +0000</pubDate>
      <link>https://dev.to/azorkai/google-crawled-230-pages-of-my-site-and-indexed-2-here-is-how-i-diagnosed-it-24i6</link>
      <guid>https://dev.to/azorkai/google-crawled-230-pages-of-my-site-and-indexed-2-here-is-how-i-diagnosed-it-24i6</guid>
      <description>&lt;p&gt;My marketing site has 230 URLs in its sitemaps. Google has indexed 2 of them.&lt;/p&gt;

&lt;p&gt;Not "pending". Not "blocked". Crawled, fetched successfully, canonical accepted, and then declined. This post is the diagnostic path I took, the two measurement mistakes I made along the way, and the Search Console API calls that finally produced a straight answer.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;The site is a Vite + React SPA prerendered at build time by Puppeteer. Every route ships as a static HTML file with the content already in the markup, served by nginx. Not SSR, but for a crawler it is indistinguishable: the body arrives full.&lt;/p&gt;

&lt;p&gt;The obvious hypotheses, in the order everyone tries them:&lt;/p&gt;

&lt;ol&gt;
&lt;li&gt;Google cannot render the JavaScript.&lt;/li&gt;
&lt;li&gt;
&lt;code&gt;robots.txt&lt;/code&gt; is blocking something.&lt;/li&gt;
&lt;li&gt;Canonicals point somewhere else.&lt;/li&gt;
&lt;li&gt;The pages are thin or duplicated.&lt;/li&gt;
&lt;li&gt;There is a manual action.&lt;/li&gt;
&lt;/ol&gt;

&lt;p&gt;All five were wrong. Here is how each died.&lt;/p&gt;

&lt;h2&gt;
  
  
  Killing the easy hypotheses
&lt;/h2&gt;

&lt;p&gt;Fetch the page as Googlebot and check the markup directly:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight shell"&gt;&lt;code&gt;curl &lt;span class="nt"&gt;-sS&lt;/span&gt; &lt;span class="nt"&gt;-A&lt;/span&gt; &lt;span class="s2"&gt;"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"&lt;/span&gt; &lt;span class="se"&gt;\&lt;/span&gt;
  https://example.com/ &lt;span class="nt"&gt;-o&lt;/span&gt; home.html

&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="s1"&gt;'&amp;lt;title&amp;gt;[^&amp;lt;]*&amp;lt;/title&amp;gt;'&lt;/span&gt; home.html
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-o&lt;/span&gt; &lt;span class="s1"&gt;'&amp;lt;link rel="canonical"[^&amp;gt;]*&amp;gt;'&lt;/span&gt; home.html
&lt;span class="nb"&gt;grep&lt;/span&gt; &lt;span class="nt"&gt;-c&lt;/span&gt; &lt;span class="s1"&gt;'&amp;lt;h1'&lt;/span&gt; home.html
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;381 KB of HTML, one &lt;code&gt;&amp;lt;h1&amp;gt;&lt;/code&gt;, correct canonical, &lt;code&gt;index, follow&lt;/code&gt;. Rendering was never the problem.&lt;/p&gt;

&lt;p&gt;For internal links, count real anchors rather than trusting a framework's router:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="n"&gt;hrefs&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;re&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;findall&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sa"&gt;r&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;&amp;lt;a\s[^&amp;gt;]*href=&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;([^&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="s"&gt;]+)&lt;/span&gt;&lt;span class="sh"&gt;"'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;internal&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;hrefs&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;h&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;startswith&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;)]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;internal&lt;/span&gt;&lt;span class="p"&gt;)),&lt;/span&gt; &lt;span class="n"&gt;html&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;count&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;nofollow&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;))&lt;/span&gt;
&lt;span class="c1"&gt;# 84 unique internal links, 0 nofollow
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;That matters because a React app that navigates with &lt;code&gt;onClick&lt;/code&gt; handlers instead of &lt;code&gt;&amp;lt;a href&amp;gt;&lt;/code&gt; is invisible to a crawler's link graph. This one was fine.&lt;/p&gt;

&lt;h2&gt;
  
  
  The Search Console API is the only source of truth
&lt;/h2&gt;

&lt;p&gt;Everything above tells you what &lt;em&gt;you&lt;/em&gt; serve. It says nothing about what Google &lt;em&gt;decided&lt;/em&gt;. For that there are exactly two useful endpoints.&lt;/p&gt;

&lt;p&gt;&lt;strong&gt;URL Inspection&lt;/strong&gt; — per-URL verdict, up to 2000 calls/day:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;googleapiclient.discovery&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;build&lt;/span&gt;

&lt;span class="n"&gt;service&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;build&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;searchconsole&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;v1&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;credentials&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="n"&gt;creds&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="n"&gt;result&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;service&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;urlInspection&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;index&lt;/span&gt;&lt;span class="p"&gt;().&lt;/span&gt;&lt;span class="nf"&gt;inspect&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;body&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="p"&gt;{&lt;/span&gt;
    &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;inspectionUrl&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;https://example.com/pricing&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
    &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;siteUrl&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;:&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;sc-domain:example.com&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
&lt;span class="p"&gt;}).&lt;/span&gt;&lt;span class="nf"&gt;execute&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;

&lt;span class="n"&gt;status&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;result&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;inspectionResult&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;][&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;indexStatusResult&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;verdict&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;|&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;coverageState&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;robotsTxtState&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;pageFetchState&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;],&lt;/span&gt; &lt;span class="n"&gt;status&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="s"&gt;lastCrawlTime&lt;/span&gt;&lt;span class="sh"&gt;'&lt;/span&gt;&lt;span class="p"&gt;])&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Run it over a representative sample rather than one URL. Mine came back like this:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;/                      PASS    | Submitted and indexed
/pricing               NEUTRAL | Crawled - currently not indexed
/features              NEUTRAL | Crawled - currently not indexed
/telegram-crm          NEUTRAL | Crawled - currently not indexed
/compare/x-vs-y        NEUTRAL | Crawled - currently not indexed
/guides/some-guide     NEUTRAL | Crawled - currently not indexed
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every one of them: &lt;code&gt;robotsTxtState: ALLOWED&lt;/code&gt;, &lt;code&gt;pageFetchState: SUCCESSFUL&lt;/code&gt;, &lt;code&gt;lastCrawlTime&lt;/code&gt; within the last two weeks, &lt;code&gt;googleCanonical&lt;/code&gt; matching &lt;code&gt;userCanonical&lt;/code&gt;.&lt;/p&gt;

&lt;p&gt;That combination is unambiguous. Google arrived, fetched a 200, parsed the page, agreed with my canonical, and chose not to index. There is no technical defect to fix, because nothing failed.&lt;/p&gt;

&lt;p&gt;The aggregate view confirmed it: &lt;strong&gt;2 indexed, 185 not indexed&lt;/strong&gt; — 99 "Discovered, currently not indexed" and 86 "Crawled, currently not indexed".&lt;/p&gt;

&lt;h2&gt;
  
  
  Two measurement mistakes worth stealing
&lt;/h2&gt;

&lt;h3&gt;
  
  
  1. &lt;code&gt;quick_ratio&lt;/code&gt; will tell you your pages are duplicates when they are not
&lt;/h3&gt;

&lt;p&gt;Template-generated pages (&lt;code&gt;/compare/a-vs-b&lt;/code&gt;, &lt;code&gt;/use-cases/industry&lt;/code&gt;) are the obvious suspects for thin content. So I measured similarity:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;difflib&lt;/span&gt;
&lt;span class="n"&gt;ratio&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="n"&gt;difflib&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nc"&gt;SequenceMatcher&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="bp"&gt;None&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page_a&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;page_b&lt;/span&gt;&lt;span class="p"&gt;).&lt;/span&gt;&lt;span class="nf"&gt;quick_ratio&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="c1"&gt;# 95.9%, 90.9%, 88.5% ...
&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Damning, apparently. Except &lt;code&gt;quick_ratio()&lt;/code&gt; compares &lt;strong&gt;character frequency multisets&lt;/strong&gt;, not sequences. Two unrelated English documents of similar length score high on it by construction. It is a cheap upper bound designed to skip expensive comparisons, not a similarity metric.&lt;/p&gt;

&lt;p&gt;The honest measurement is to strip the shared chrome and count what survives:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight python"&gt;&lt;code&gt;&lt;span class="kn"&gt;from&lt;/span&gt; &lt;span class="n"&gt;collections&lt;/span&gt; &lt;span class="kn"&gt;import&lt;/span&gt; &lt;span class="n"&gt;Counter&lt;/span&gt;

&lt;span class="c1"&gt;# how many pages does each line of text appear on?
&lt;/span&gt;&lt;span class="n"&gt;counts&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nc"&gt;Counter&lt;/span&gt;&lt;span class="p"&gt;()&lt;/span&gt;
&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;values&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;line&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="nf"&gt;set&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;):&lt;/span&gt;
        &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;line&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;+=&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;

&lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;docs&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;items&lt;/span&gt;&lt;span class="p"&gt;():&lt;/span&gt;
    &lt;span class="n"&gt;unique&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt; &lt;span class="k"&gt;if&lt;/span&gt; &lt;span class="n"&gt;counts&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt; &lt;span class="o"&gt;==&lt;/span&gt; &lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;
    &lt;span class="n"&gt;total_w&lt;/span&gt;  &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;lines&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="n"&gt;unique_w&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="nf"&gt;sum&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="nf"&gt;len&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;l&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="nf"&gt;split&lt;/span&gt;&lt;span class="p"&gt;())&lt;/span&gt; &lt;span class="k"&gt;for&lt;/span&gt; &lt;span class="n"&gt;l&lt;/span&gt; &lt;span class="ow"&gt;in&lt;/span&gt; &lt;span class="n"&gt;unique&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
    &lt;span class="nf"&gt;print&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;url&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="sa"&gt;f&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;unique_w&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;/&lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;total_w&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt; = &lt;/span&gt;&lt;span class="si"&gt;{&lt;/span&gt;&lt;span class="n"&gt;unique_w&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;&lt;span class="mi"&gt;100&lt;/span&gt;&lt;span class="o"&gt;//&lt;/span&gt;&lt;span class="n"&gt;total_w&lt;/span&gt;&lt;span class="si"&gt;}&lt;/span&gt;&lt;span class="s"&gt;% unique&lt;/span&gt;&lt;span class="sh"&gt;"&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Result: &lt;strong&gt;90-95% unique per page&lt;/strong&gt;, 1400-1950 words of it, with only ~35 words of shared boilerplate (nav, CTA strip, footer). The pages were not the problem. Had I stopped at &lt;code&gt;quick_ratio&lt;/code&gt; I would have spent a week rewriting content that was already fine.&lt;/p&gt;

&lt;h3&gt;
  
  
  2. &lt;code&gt;site:&lt;/code&gt; in a search box is not an index count
&lt;/h3&gt;

&lt;p&gt;&lt;code&gt;num=100&lt;/code&gt; no longer works on Google. If you count results on the first page you are counting ten. Paginate with &lt;code&gt;start=&lt;/code&gt; and stop when a page returns fewer than a full set, or just read the Page Indexing report, which is authoritative.&lt;/p&gt;

&lt;h2&gt;
  
  
  What the answer actually was
&lt;/h2&gt;

&lt;p&gt;With the technical hypotheses dead, one report explained everything:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;External links: Total 10   (3 domains)
Internal links: Total 0
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Ten inbound links from three domains, all to the homepage. Meanwhile a backlink tool reported 197 links — but more than half came from a single PBN spam network hammering one keyword-stuffed anchor. Google had simply not counted them. The tool's number was noise; Search Console's was the signal.&lt;/p&gt;

&lt;p&gt;So: a site with real content, clean markup, correct canonicals, valid sitemaps, no manual action, and effectively zero independent references, publishing 230 pages into one of the most saturated categories on the web.&lt;/p&gt;

&lt;p&gt;"Crawled, currently not indexed" is not a bug report. It is a verdict. Google read the pages and decided the index did not need them. More crawl budget does not fix that, and neither does resubmitting a sitemap — I have two months of flat data to prove it.&lt;/p&gt;

&lt;h2&gt;
  
  
  The part that generalises
&lt;/h2&gt;

&lt;ul&gt;
&lt;li&gt;If &lt;code&gt;pageFetchState&lt;/code&gt; is &lt;code&gt;SUCCESSFUL&lt;/code&gt; and &lt;code&gt;robotsTxtState&lt;/code&gt; is &lt;code&gt;ALLOWED&lt;/code&gt;, stop debugging your stack. The answer is not in your code.&lt;/li&gt;
&lt;li&gt;Check the Links report before rewriting content. Zero external links explains more indexing problems than any rendering bug.&lt;/li&gt;
&lt;li&gt;Backlink tools and Search Console disagree, and for "does Google count this link", Search Console wins.&lt;/li&gt;
&lt;li&gt;Verify a claimed fix against production before repeating it. Three broken URLs I was still citing from a month-old crawl had already been redirected; I only found out because I re-checked with &lt;code&gt;curl&lt;/code&gt; instead of trusting my notes.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;The uncomfortable conclusion is that for a new domain in a crowded category, indexing is downstream of being referenced by someone other than yourself. That is a distribution problem wearing a technical costume.&lt;/p&gt;




&lt;p&gt;&lt;em&gt;I work on &lt;a href="https://pinlyx.com" rel="noopener noreferrer"&gt;Pinlyx&lt;/a&gt;, an omnichannel CRM (it was called CRM Solid when this was written). The numbers above are from its marketing site, which is a live and ongoing example of the problem.&lt;/em&gt;&lt;/p&gt;

</description>
      <category>seo</category>
      <category>webdev</category>
      <category>googlecloud</category>
      <category>python</category>
    </item>
    <item>
      <title>Four traps in querying 1.79M Overture Maps records with DuckDB</title>
      <dc:creator>Emirhan Güven</dc:creator>
      <pubDate>Tue, 01 Sep 2026 10:18:27 +0000</pubDate>
      <link>https://dev.to/azorkai/four-traps-in-querying-179m-overture-maps-records-with-duckdb-4231</link>
      <guid>https://dev.to/azorkai/four-traps-in-querying-179m-overture-maps-records-with-duckdb-4231</guid>
      <description>&lt;p&gt;We needed a business directory for Turkey inside our CRM, so we went to&lt;br&gt;
&lt;a href="https://overturemaps.org/" rel="noopener noreferrer"&gt;Overture Maps&lt;/a&gt; places: open, CDLA-Permissive, no API key,&lt;br&gt;
no quota, ~1.8M Turkish records sitting in public GeoParquet on S3.&lt;/p&gt;

&lt;p&gt;Reading it turned out to be the easy part. Here are the four things that actually cost us&lt;br&gt;
time, and the numbers that came out the other end.&lt;/p&gt;
&lt;h2&gt;
  
  
  1. A join against remote parquet does not stream
&lt;/h2&gt;

&lt;p&gt;The obvious query is one statement: read the places theme, read the divisions theme, keep&lt;br&gt;
the rows whose point falls inside a Turkish province.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="c1"&gt;-- Do not do this against S3.&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;read_parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'s3://.../theme=places/*/*'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hive_partitioning&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;
&lt;span class="k"&gt;JOIN&lt;/span&gt; &lt;span class="n"&gt;regions&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt; &lt;span class="k"&gt;ON&lt;/span&gt; &lt;span class="n"&gt;ST_Within&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;geometry&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;geom&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;country&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'TR'&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;DuckDB materialises that join before it emits a single row. Ours sat at &lt;strong&gt;2.5 GB of RAM&lt;br&gt;
and produced nothing after fifteen minutes&lt;/strong&gt;, with no way to tell whether it was making&lt;br&gt;
progress.&lt;/p&gt;

&lt;p&gt;The fix is boring and works: &lt;strong&gt;two stages&lt;/strong&gt;. Stage one pulls the country down with a plain&lt;br&gt;
bounding box and zero spatial work.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TEMP&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;places_tr&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;id&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="k"&gt;names&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;primary&lt;/span&gt;        &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;categories&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;primary&lt;/span&gt;   &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;category&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;websites&lt;/span&gt;&lt;span class="p"&gt;[&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;]&lt;/span&gt;          &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;website&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;bbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ymin&lt;/span&gt;            &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;lat&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;bbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;xmin&lt;/span&gt;            &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;lon&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt;
       &lt;span class="n"&gt;geometry&lt;/span&gt;             &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;geom&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;read_parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'s3://.../theme=places/*/*'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hive_partitioning&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;bbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;xmin&lt;/span&gt; &lt;span class="k"&gt;BETWEEN&lt;/span&gt; &lt;span class="mi"&gt;25&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;5&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="mi"&gt;45&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;0&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;bbox&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;ymin&lt;/span&gt; &lt;span class="k"&gt;BETWEEN&lt;/span&gt; &lt;span class="mi"&gt;35&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;7&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="mi"&gt;42&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="mi"&gt;3&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;names&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;primary&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
  &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;operating_status&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt; &lt;span class="k"&gt;OR&lt;/span&gt; &lt;span class="n"&gt;operating_status&lt;/span&gt; &lt;span class="o"&gt;&amp;lt;&amp;gt;&lt;/span&gt; &lt;span class="s1"&gt;'closed'&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;A rectangle over Turkey also sweeps in slivers of Greece, Bulgaria and Georgia. That is&lt;br&gt;
fine, stage two throws them out. Stage two then loops province by province, so each join&lt;br&gt;
touches one polygon and a few thousand candidate rows:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="o"&gt;*&lt;/span&gt; &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;places_tr&lt;/span&gt; &lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;tr_regions&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;
&lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;region_name&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'İzmir'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;ST_Within&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;p&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;geom&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;r&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="n"&gt;geom&lt;/span&gt;&lt;span class="p"&gt;);&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Every step is bounded and you can watch it progress. Same result, and the whole run&lt;br&gt;
finishes instead of hanging.&lt;/p&gt;
&lt;h2&gt;
  
  
  2. 81 provinces, 109 polygons
&lt;/h2&gt;

&lt;p&gt;Turkey has 81 provinces. Overture's divisions theme hands you &lt;strong&gt;109&lt;/strong&gt; &lt;code&gt;region&lt;/code&gt; rows for the&lt;br&gt;
country. My first reading was "duplicates": Adana, Antalya and Artvin each arrive twice, so&lt;br&gt;
merge by name and move on. That works, but it hides the real reason, which is in the schema:&lt;br&gt;
every coastal province has a second &lt;code&gt;division_area&lt;/code&gt; row with &lt;code&gt;class = 'maritime'&lt;/code&gt; for its&lt;br&gt;
territorial waters (81 &lt;code&gt;land&lt;/code&gt; + 28 &lt;code&gt;maritime&lt;/code&gt; in release 2026-07-22.0; Antalya's maritime&lt;br&gt;
polygon is larger than its land). If you loop over the raw rows you process those provinces&lt;br&gt;
(and their companies) twice.&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight sql"&gt;&lt;code&gt;&lt;span class="k"&gt;CREATE&lt;/span&gt; &lt;span class="k"&gt;TEMP&lt;/span&gt; &lt;span class="k"&gt;TABLE&lt;/span&gt; &lt;span class="n"&gt;tr_regions&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt;
&lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="n"&gt;region_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;ST_Union_Agg&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="n"&gt;geom&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;geom&lt;/span&gt;
&lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="p"&gt;(&lt;/span&gt;
  &lt;span class="k"&gt;SELECT&lt;/span&gt; &lt;span class="k"&gt;names&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;primary&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;region_name&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;geometry&lt;/span&gt; &lt;span class="k"&gt;AS&lt;/span&gt; &lt;span class="n"&gt;geom&lt;/span&gt;
  &lt;span class="k"&gt;FROM&lt;/span&gt; &lt;span class="n"&gt;read_parquet&lt;/span&gt;&lt;span class="p"&gt;(&lt;/span&gt;&lt;span class="s1"&gt;'.../theme=divisions/*/*'&lt;/span&gt;&lt;span class="p"&gt;,&lt;/span&gt; &lt;span class="n"&gt;hive_partitioning&lt;/span&gt;&lt;span class="o"&gt;=&lt;/span&gt;&lt;span class="mi"&gt;1&lt;/span&gt;&lt;span class="p"&gt;)&lt;/span&gt;
  &lt;span class="k"&gt;WHERE&lt;/span&gt; &lt;span class="n"&gt;country&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'TR'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="n"&gt;subtype&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'region'&lt;/span&gt; &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;class&lt;/span&gt; &lt;span class="o"&gt;=&lt;/span&gt; &lt;span class="s1"&gt;'land'&lt;/span&gt;
    &lt;span class="k"&gt;AND&lt;/span&gt; &lt;span class="k"&gt;names&lt;/span&gt;&lt;span class="p"&gt;.&lt;/span&gt;&lt;span class="k"&gt;primary&lt;/span&gt; &lt;span class="k"&gt;IS&lt;/span&gt; &lt;span class="k"&gt;NOT&lt;/span&gt; &lt;span class="k"&gt;NULL&lt;/span&gt;
&lt;span class="p"&gt;)&lt;/span&gt;
&lt;span class="k"&gt;GROUP&lt;/span&gt; &lt;span class="k"&gt;BY&lt;/span&gt; &lt;span class="n"&gt;region_name&lt;/span&gt;&lt;span class="p"&gt;;&lt;/span&gt;
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;&lt;code&gt;class = 'land'&lt;/code&gt; is the fix; the &lt;code&gt;GROUP BY&lt;/code&gt; stays as a guard so downstream code can rely on&lt;br&gt;
exactly one polygon per province name. One line, and it removes a whole class of&lt;br&gt;
double-counting bugs.&lt;/p&gt;

&lt;h2&gt;
  
  
  3. The address field will not tell you the province
&lt;/h2&gt;

&lt;p&gt;This is the one worth knowing before you plan anything. Overture's address &lt;code&gt;region&lt;/code&gt; field&lt;br&gt;
is &lt;strong&gt;null on roughly 92% of Turkish records&lt;/strong&gt;, and inconsistent on the rest. There is no&lt;br&gt;
salvaging it. Province has to be assigned geographically, from coordinates.&lt;/p&gt;

&lt;p&gt;And it has to be a real point-in-polygon test, not a bounding box. Province bounding boxes&lt;br&gt;
overlap heavily; picking the smallest containing box puts &lt;strong&gt;central İzmir in Manisa&lt;/strong&gt; and&lt;br&gt;
&lt;strong&gt;central Antalya in Burdur&lt;/strong&gt;. We found that the entertaining way.&lt;/p&gt;

&lt;h2&gt;
  
  
  4. Lone surrogates will abort your bulk insert
&lt;/h2&gt;

&lt;p&gt;Loading the results into Postgres with Npgsql's binary &lt;code&gt;COPY&lt;/code&gt;, the batch died on real&lt;br&gt;
company names. The cause: &lt;strong&gt;lone surrogates&lt;/strong&gt;, half of an emoji pair, usually from a&lt;br&gt;
truncated source record. Npgsql's UTF-8 encoder throws on those and takes the whole batch&lt;br&gt;
with it.&lt;/p&gt;

&lt;p&gt;Two details that cost us a second round trip:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Truncating a string to a column width can &lt;em&gt;split&lt;/em&gt; a valid surrogate pair, so sanitising
has to run &lt;strong&gt;after&lt;/strong&gt; truncation, not before.&lt;/li&gt;
&lt;li&gt;Null bytes need dropping outright; Postgres rejects them in text columns.&lt;/li&gt;
&lt;/ul&gt;

&lt;h2&gt;
  
  
  What it bought
&lt;/h2&gt;

&lt;p&gt;Querying Overture's S3 parquet directly from a Turkish host: &lt;strong&gt;277 seconds&lt;/strong&gt; for a single&lt;br&gt;
province. The same query against the local Postgres catalogue: &lt;strong&gt;15 ms&lt;/strong&gt;. That is the&lt;br&gt;
difference between a feature and a batch job.&lt;/p&gt;

&lt;h2&gt;
  
  
  And then the actual finding
&lt;/h2&gt;

&lt;p&gt;With 1,786,700 businesses loaded, we asked how many of them have a website. The field says&lt;br&gt;
&lt;strong&gt;684,496, or 38.3%&lt;/strong&gt;.&lt;/p&gt;

&lt;p&gt;So we requested every one of those addresses. &lt;strong&gt;413,777 answered.&lt;/strong&gt; The rest time out,&lt;br&gt;
refuse the connection, or sit on an expired domain.&lt;/p&gt;

&lt;p&gt;The honest national figure is &lt;strong&gt;23.2%&lt;/strong&gt;: roughly one Turkish business in four can be&lt;br&gt;
reached on the web at all. Meanwhile 542,323 businesses publish a phone number and no&lt;br&gt;
website whatsoever.&lt;/p&gt;

&lt;p&gt;The same gap shows up in the socials array, in the other direction: Overture has Facebook&lt;br&gt;
on 95.3% of Turkish records but Instagram on only ~14k, an artefact of when the source&lt;br&gt;
listings were assembled. Crawling the company websites we could reach found Instagram on&lt;br&gt;
about 36% of them, taking the count to 232,928. &lt;strong&gt;Treat a populated field as a claim, not&lt;br&gt;
a measurement.&lt;/strong&gt;&lt;/p&gt;

&lt;h2&gt;
  
  
  The data is open
&lt;/h2&gt;

&lt;p&gt;Everything above is published under CC BY 4.0, province and sector breakdowns included:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;Report and tables: &lt;a href="https://pinlyx.com/research/turkey-business-digital-report" rel="noopener noreferrer"&gt;pinlyx.com/research/turkey-business-digital-report&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;CSV, JSON and the SQL that produces the figures: &lt;a href="https://github.com/Pinlyx/turkey-business-digital-data" rel="noopener noreferrer"&gt;github.com/Pinlyx/turkey-business-digital-data&lt;/a&gt;
&lt;/li&gt;
&lt;li&gt;Archived with a DOI: &lt;a href="https://doi.org/10.5281/zenodo.22217770" rel="noopener noreferrer"&gt;10.5281/zenodo.22217770&lt;/a&gt;
&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;If you are working with Overture in another country, I would genuinely like to know&lt;br&gt;
whether the &lt;code&gt;region&lt;/code&gt; field is as empty there as it is here.&lt;/p&gt;

</description>
      <category>duckdb</category>
      <category>sql</category>
      <category>opendata</category>
      <category>dotnet</category>
    </item>
  </channel>
</rss>
