Is cross-posting duplicate content? I measured the overlap first

#seo #webdev #writing #javascript

If you syndicate the same posts to dev.to, Hashnode, Zenn or Qiita, sooner or later you wonder whether search engines read the copies as duplicate content. I got far enough into that worry to write a remediation plan: unpublish the live copies, set canonical everywhere, start over.

Then I counted the actual overlap. Across 11 articles and 22 files, three prose lines matched between origin and copy — and two of them were --- rules. The plan was unnecessary. What replaced it was adding a canonical URL on the platforms that accept one.

Here is the measurement, and the support matrix I wish I had looked up first.

Count the overlap by line, then split it by kind

Three steps:

strip frontmatter
normalize whitespace, drop blank lines
tag each line by whether it sits inside a code fence

Step 3 is the one that decides the answer. Without it you get a single percentage and no way to interpret it.

import fs from "node:fs";

const norm = (s) => s.replace(/\s+/g, " ").trim();

function bodyOf(file) {
  let t = fs.readFileSync(file, "utf8");
  if (t.startsWith("---")) t = t.slice(t.indexOf("\n---", 3) + 4); // strip frontmatter
  return t;
}

// Split into [normalized line, kind]
function classify(text) {
  const out = [];
  let inFence = false;
  for (const raw of text.split("\n")) {
    const line = norm(raw);
    if (/^```/.test(line)) { inFence = !inFence; out.push([line, "code"]); continue; }
    if (!line) continue;
    if (inFence) { out.push([line, "code"]); continue; }
    if (/^#{1,6}\s/.test(line)) { out.push([line, "heading"]); continue; }
    if (/^\|/.test(line)) { out.push([line, "table"]); continue; }
    out.push([line, "prose"]);
  }
  return out;
}

// Put the origin's lines in a set, walk the syndicated copy
const origin = new Set(classify(bodyOf(originFile)).map(([l]) => l));
const ext = classify(bodyOf(copyFile));
const matched = ext.filter(([l]) => origin.has(l));

const count = (k) => matched.filter(([, t]) => t === k).length;
console.log({
  ext: ext.length,
  matched: matched.length,
  code: count("code"),
  table: count("table"),
  heading: count("heading"),
  prose: count("prose"),
});

Skip the inFence toggle and classify on the leading character alone, and shell comments starting with #, plus YAML and SQL lines starting with |, all land in the prose bucket. That bucket is the number you are trying to trust, so it has to be clean.

Blank lines matter for the same reason: they always match, so keeping them inflates both sides of the ratio.

The result: 762 of the 795 matching lines were code

Run across my 11 articles, counted as 22 pairs because English and Japanese are separate files:

Metric	Value
Lines in the syndicated bodies	1,658
Lines matching the origin	795 (47.9%)
— inside code fences	762
— table rows	13
— headings	17
— prose	3
Prose lines in the syndicated bodies	393
Prose overlap	0.8%

Printing the three matching prose lines: two --- rules and one English bullet.

Overlap tracks code density, not copying. The posts with the most snippets score highest, which makes the raw percentage actively misleading. Code repeats because both versions describe the same implementation — rewriting a CREATE UNIQUE INDEX statement for the copy would just make one of them wrong.

If you write each version separately rather than pasting, run this before deciding anything.

Where you can declare a canonical URL, and where you cannot

Support is uneven. The API payloads tell you quickly:

Platform	Canonical	Where it goes
dev.to	Yes	`article.canonical_url` on the articles endpoint
Hashnode	Yes	`originalArticleURL` on the `publishPost` mutation
Medium	Yes	`canonicalUrl` on the posts endpoint, or Import from URL
Qiita	No	no equivalent parameter on the items API
Zenn	No	no such key in article frontmatter
note	No	no field in the editor

The Forem API exposes canonical_url on the article object, so it is one line in the payload:

body: JSON.stringify({
  article: {
    title,
    body_markdown: body,
    published: false,
    tags,
    canonical_url: "https://example.com/blog/<slug>/",
  },
}),

Qiita's API v2 takes title, body, tags, private, tweet and organization_url_name for item creation — nothing that points at an external original. Zenn's article frontmatter is title, emoji, type, topics, published, publication_name, same situation.

Gotcha: Hashnode drafts cannot carry a canonical

originalArticleURL exists on the publishPost input but not on createDraft. Create a draft, hit publish in the dashboard, and the canonical never gets attached.

const input = publish
  ? { title, contentMarkdown, publicationId, tags, originalArticleURL: canonical }
  : { title, contentMarkdown, publicationId, tags };   // no field for it on a draft

dev.to fixes the value at post time too, so already-published articles need PUT /articles/{id} or a manual edit in the editor.

On some platforms, "temporarily unpublish" does not exist

Without a canonical field, the way back from unpublishing is posting again: new URL, views reset, inbound links broken. It looks reversible and is not.

Which is the practical reason to measure first. Thirty lines of script, one run, and the answer may well delete the entire task list behind it.

The full story — how I ended up conflating an AdSense decision with search-side duplicate handling, the per-channel traffic numbers behind the reversal, and the per-article overlap breakdown — is on Aulvem → Aulvem | Is cross-posting duplicate content? I measured it before deleting anything

DEV Community