DEV Community

Cover image for One curl Script, Three Sites, Three Different Ways to Handle Duplicate URLs
Hannan Shoukat for 8Bit Enterprise

Posted on AI-assisted

One curl Script, Three Sites, Three Different Ways to Handle Duplicate URLs

Most sites answer on more URLs than their owners think. http:// and https://, with and without www, with and without a trailing slash, in uppercase, with a ?utm_source= tag from a newsletter. To a browser, they're the same page. To a search engine, each one is a separate URL it has to group.

I wanted a quick way to see how a page handles all of them, so I wrote a small script.

The script

It takes one URL, builds five common variants, and prints either the redirect target or the canonical tag for each.

#!/usr/bin/env bash
# Check how one page handles common duplicate URL variants.
# Usage: ./dupcheck.sh https://example.com/some-page/
url="$1"
tmp=$(mktemp); trap 'rm -f "$tmp"' EXIT
scheme="${url%%://*}"; rest="${url#*://}"
host="${rest%%/*}"; path="/${rest#*/}"; [ "$rest" = "$host" ] && path="/"

if [[ "$host" == www.* ]]; then alt_host="${host#www.}"; else alt_host="www.$host"; fi
if [[ "$path" == */ && "$path" != "/" ]]; then alt_path="${path%/}"; else alt_path="$path/"; fi
# uppercase only the last path segment, e.g. /blog/My-Post/ -> /blog/MY-POST/
trim="${path%/}"; last="${trim##*/}"; base="${trim%/*}"
upper_path="$base/$(echo "$last" | tr '[:lower:]' '[:upper:]')"; [[ "$path" == */ ]] && upper_path="$upper_path/"

check() {
  local label="$1" u="$2"
  local out code loc canon
  out=$(curl -s -A "dupcheck" -o "$tmp" -w '%{http_code} %{redirect_url}' "$u")
  code="${out%% *}"; loc="${out#* }"
  canon=$(grep -oiE '<link[^>]+rel=.canonical.[^>]*>' "$tmp" | grep -oiE 'href=.[^"'"'"' >]+' | head -1 | cut -c7-)
  if [[ "$code" == 000 ]]; then printf '%-16s no response\n' "$label"
  elif [[ "$code" == 30* ]]; then printf '%-16s %s -> %s\n' "$label" "$code" "$loc"
  else printf '%-16s %s  canonical: %s\n' "$label" "$code" "${canon:-NONE}"; fi
}

check "original"      "$url"
check "other scheme"  "$([ "$scheme" = https ] && echo http || echo https)://$host$path"
check "www toggle"    "$scheme://$alt_host$path"
check "slash toggle"  "$scheme://$host$alt_path"
check "uppercase"     "$scheme://$host$upper_path"
check "utm param"     "$url$([[ "$url" == *\?* ]] && echo '&' || echo '?')utm_source=dupcheck"
Enter fullscreen mode Exit fullscreen mode

Save it as dupcheck.sh, run chmod +x dupcheck.sh, then ./dupcheck.sh https://yoursite.com/some-page/.

Three sites, three answers

I ran it on 18 November 2026 against three very different stacks.

Our WordPress blog (Yoast SEO):

original         200  canonical: https://8bitenterprise.com/blog/schema-markup/
other scheme     301 -> https://8bitenterprise.com/blog/schema-markup/
www toggle       301 -> https://8bitenterprise.com/blog/schema-markup/
slash toggle     301 -> https://8bitenterprise.com/blog/schema-markup/
uppercase        200  canonical: https://8bitenterprise.com/blog/schema-markup/
utm param        200  canonical: https://8bitenterprise.com/blog/schema-markup/
Enter fullscreen mode Exit fullscreen mode

Shopify's Dawn demo store (a product page):

original         200  canonical: https://theme-dawn-demo.myshopify.com/products/puff-olive-leaf
other scheme     301 -> https://theme-dawn-demo.myshopify.com/products/puff-olive-leaf
www toggle       no response
slash toggle     200  canonical: https://theme-dawn-demo.myshopify.com/products/puff-olive-leaf
uppercase        200  canonical: https://theme-dawn-demo.myshopify.com/products/puff-olive-leaf
utm param        200  canonical: https://theme-dawn-demo.myshopify.com/products/puff-olive-leaf
Enter fullscreen mode Exit fullscreen mode

(The www line fails because a .myshopify.com subdomain has no www version. A store on its own domain would answer.)

DEV itself (a profile page):

original         200  canonical: https://dev.to/hannanshoukat
other scheme     301 -> https://dev.to/hannanshoukat
www toggle       301 -> https://dev.to/hannanshoukat
slash toggle     200  canonical: https://dev.to/hannanshoukat
uppercase        301 -> https://dev.to/hannanshoukat
utm param        200  canonical: https://dev.to/hannanshoukat
Enter fullscreen mode Exit fullscreen mode

All three end up pointing at one URL, but they disagree on how:

  • Trailing slash: WordPress redirects it. Shopify and DEV load the page and rely on the canonical tag.
  • Uppercase: DEV redirects it. WordPress and Shopify load it and rely on the canonical tag.
  • Tracking parameters: all three load the page with a canonical tag, which makes sense because the parameter has to reach analytics.

Does it matter which one you pick?

According to Google, both work. Its canonicalization docs call redirects and rel="canonical" both "a strong signal," while a sitemap is "a weak signal." The difference is that a canonical tag is a hint Google can overrule, and a redirect means the duplicate URL is never served at all.

What does matter is getting no signal: a variant that returns 200 with canonical: NONE. That's the line to look for.

I also tested Shopify's collection, variant and filter URLs, plus what Search Console's duplicate statuses mean. That's in a longer write-up on duplicate content and product pages if you're working on a store.

Over to you

  • Run it on your own site. Does any line come back 200 canonical: NONE?
  • Trailing slashes: do you redirect them or rely on the canonical tag? Has either ever caused you a problem?
  • Is there a variant I've missed that you check on client sites? I'd like to add it.

Top comments (0)