DEV Community: mattleeee

Three-Layer Validation Before You Trust a Backtest

mattleeee — Mon, 26 Oct 2026 01:02:11 +0000

Why One Good Backtest Means Nothing

A single equity curve is a sample of size one. Optimize hard enough over enough combinations and something will look great by luck alone. The question is not "does this rule make money in the backtest?" but "does this rule make money for reasons that survive contact with slightly different assumptions?"

The gate I use before any rule reaches the live scanner has three layers: a parameter sweep, a monthly consistency check, and a component ablation. Each layer answers a different question, and each has a hard pass/fail threshold. A rule has to clear all three. This post walks through the mechanics, then through a real rejection: a MACD divergence entry that passed layer 1 and died on layers 2 and 3.

Layer 1: Parameter Sweep

The question it answers

Is the edge a plateau or a spike? If performance only exists at fast=12, slow=26, signal=9, you have found noise, not a rule. Neighboring parameter values should produce comparable results, and the chosen point should sit near the center of a profitable region, not on its edge.

The metrics

For each parameter combination in the grid, record:

Trade count — anything under ~50 is too thin to say much.
Profit factor (PF) — gross wins / gross losses.
Expectancy per trade — in R units (risk multiples), so results are comparable across parameter sets.
Max drawdown — in R, not percent, so it doesn't depend on position sizing.
Plateau score — the fraction of neighboring combinations (within one step in each dimension) that stay above the PF threshold.

The thresholds

Median PF across the sweep ≥ 1.3
At least 70% of combinations with ≥ 50 trades have PF > 1.0
Plateau score at the chosen point ≥ 0.6
Chosen point is not on the grid boundary

The code

import itertools
import numpy as np
import pandas as pd

def sweep(params_grid, run_backtest, min_trades=50):
    rows = []
    for combo in itertools.product(*params_grid.values()):
        params = dict(zip(params_grid.keys(), combo))
        res = run_backtest(**params)
        if res["trades"] < min_trades:
            continue
        rows.append({**params, **res})
    return pd.DataFrame(rows)

def plateau_score(df, param_names, chosen, pf_col="pf", threshold=1.0):
    """Fraction of 1-step neighbors of `chosen` with PF above threshold."""
    neighbors = []
    for name in param_names:
        values = sorted(df[name].unique())
        idx = values.index(chosen[name])
        for offset in (-1, 1):
            j = idx + offset
            if 0 <= j < len(values):
                mask = pd.Series(True, index=df.index)
                for n in param_names:
                    target = chosen[n] if n != name else values[j]
                    mask &= df[n] == target
                if mask.any():
                    neighbors.append(df.loc[mask, pf_col].iloc[0])
    if not neighbors:
        return 0.0
    return float(np.mean(np.array(neighbors) > threshold))

grid = {
    "fast":   [8, 10, 12, 14, 16],
    "slow":   [20, 24, 26, 30, 34],
    "signal": [7, 9, 11],
}
results = sweep(grid, run_macd_divergence_backtest)

chosen = {"fast": 12, "slow": 26, "signal": 9}
score = plateau_score(results, list(grid.keys()), chosen)

print(f"median PF: {results['pf'].median():.2f}")
print(f"share PF>1: {(results['pf'] > 1.0).mean():.0%}")
print(f"plateau score at chosen point: {score:.2f}")

A plateau score of 0.6 means most one-step neighbors still clear PF 1.0. That is a low bar, deliberately. If a rule can't survive a one-step nudge, it has no business being tested further.

Layer 2: Monthly Consistency

The question it answers

Is the annual result distributed across time, or is it one monster month doing all the work? A strategy with one +40% month and eleven flat months is not a strategy. It's a lottery ticket that already paid out, and the next ticket is unlikely to pay again.

The metrics

Bucket trades by calendar month and compute:

Share of profitable months — count of months with positive PnL / total months.
Best-month contribution — best month's PnL / total PnL.
Top-3-month contribution — same, for the three best months.
Median monthly return and worst month in R.

The thresholds

≥ 55% of months positive
Best month ≤ 30% of total PnL
Top 3 months ≤ 55% of total PnL
Worst month no worse than −3R (scaled to your per-trade risk)

The best-month cap is the important one. It is the metric that catches curve-fit strategies that happen to overlap a single volatility event.

The code

def monthly_consistency(trades: pd.DataFrame, risk_per_trade=1.0):
    """trades: columns ['exit_time', 'pnl_r']"""
    monthly = (trades
               .assign(month=trades["exit_time"].dt.to_period("M"))
               .groupby("month")["pnl_r"]
               .sum()
               .sort_index())
    total = monthly.sum()
    if total <= 0:
        return {"pass": False, "reason": "non-positive total PnL"}

    best = monthly.max()
    top3 = monthly.nlargest(3).sum()
    stats = {
        "months": len(monthly),
        "positive_share": (monthly > 0).mean(),
        "best_share": best / total,
        "top3_share": top3 / total,
        "median_month": monthly.median(),
        "worst_month": monthly.min(),
    }
    stats["pass"] = (
        stats["months"] >= 24
        and stats["positive_share"] >= 0.55
        and stats["best_share"] <= 0.30
        and stats["top3_share"] <= 0.55
        and stats["worst_month"] >= -3 * risk_per_trade
    )
    return stats

The months >= 24 requirement matters. With twelve months, "positive share" is dominated by a handful of observations and the test is close to meaningless.

Layer 3: Robustness Ablation

The question it answers

Does each component of the rule contribute, and does the rule degrade gracefully when a component is removed? A robust rule loses some edge when you strip a filter. A fragile rule collapses to zero, or worse, only works with every component present — a sign the components were fitted to each other.

The metrics

For each component, run the backtest with that component removed (or replaced by a neutral version) and record:

PF delta — component PF minus full-rule PF.
Expectancy delta in R.
Retention — component expectancy / full-rule expectancy.

The thresholds

Removing any single component leaves expectancy ≥ 60% of the full rule (graceful degradation).
Removing the core component (the entry trigger itself, not a filter) drops expectancy below 50% and PF below 1.1 (the core actually matters).
No component removal improves expectancy by more than 15% (if it does, that component is hurting and should be cut).

The third criterion is the one people skip. It catches filters that were added to patch a specific losing streak in the sample and do nothing out of sample.

The code

def ablation(full_result, variants: dict, core="divergence"):
    """variants: name -> backtest result dict with 'pf' and 'expectancy_r'"""
    base_pf = full_result["pf"]
    base_exp = full_result["expectancy_r"]
    rows = []
    for name, res in variants.items():
        rows.append({
            "component": name,
            "pf": res["pf"],
            "expectancy_r": res["expectancy_r"],
            "retention": res["expectancy_r"] / base_exp if base_exp else 0.0,
            "pf_delta": res["pf"] - base_pf,
        })
    df = pd.DataFrame(rows)

    checks = {
        "graceful": (df.loc[df["component"] != core, "retention"] >= 0.60).all(),
        "core_matters": df.loc[df["component"] == core, "retention"].lt(0.50).all(),
        "no_harmful": (df["retention"] <= 1.15).all(),
    }
    return df, checks

The Case Study: A MACD Divergence Entry

The rule: enter long when price makes a lower low but MACD histogram makes a higher low (bullish divergence), confirmed by a close above the signal line. Exit on a fixed 2R target or a time stop of 20 bars. Filters included a trend filter (price above the 200-EMA), a volume filter (volume above its 20-bar average), and a volatility floor (ATR above its 20-bar median).

Layer 1: passed

The sweep over fast ∈ [8,16], slow ∈ [20,34], signal ∈ [7,11] produced 75 combinations, 68 with enough trades. Median PF was 1.41, 78% of combinations had PF above 1.0, and the plateau score at (12, 26, 9) was 0.71. The chosen point sat comfortably inside the grid. Nothing suspicious yet.

Layer 2: failed

Over 31 months, only 48% of months were positive. Worse, the best month contributed 44% of total PnL, and the top three months contributed 71%. The best month coincided with a broad volatility expansion — the rule caught one regime and did nothing in the other thirty. Median monthly PnL was slightly negative.

Verdict: fail on positive_share (0.48 < 0.55), best_share (0.44 > 0.30), and top3_share (0.71 > 0.55). This alone is enough to reject the rule.

Layer 3: failed

The ablation was run anyway, out of curiosity:

Removed	PF	Expectancy (R)	Retention
Trend filter	1.38	0.11	0.73
Volume filter	1.52	0.17	1.13
Volatility floor	1.09	0.06	0.40
Divergence (core)	0.94	−0.01	−0.07

Two problems. The volatility floor is doing most of the work — removing it drops retention to 0.40, well below the 0.60 floor. That means the "divergence" rule is largely a low-volatility regime filter wearing a costume. And removing the volume filter improves expectancy by 13%, which is under the 15% harm threshold but close enough to be a warning sign: the filter was probably added to patch a specific stretch of the sample.

The core divergence component does matter (removal sends expectancy negative), but that doesn't save the rule. A rule whose edge lives mostly in a regime filter, and whose only positive months come from one volatility event, is not a rule. It's a description of one period.

The rejection

Three-layer result: pass, fail, fail. The rule went into the rejected list with the metrics recorded, so future me doesn't re-test it in six months and get excited all over again. That bookkeeping is half the value of the process.

Practical Notes

Run the layers in order. Layer 1 is cheap and filters out most junk. Layer 2 is cheaper than layer 3 and catches the most common failure mode. Don't run ablations on rules that can't pass a monthly check.
Log every metric, including failures. A rejection is only useful if it's documented. Keep the sweep DataFrame and the ablation table next to the rule definition.
Thresholds are conventions, not laws. The numbers above are tuned to my trade frequency and risk unit. If your rules trade twenty times a year, months >= 24 is too strict; if they trade daily, best_share <= 0.30 is probably too loose. Pick thresholds once, write them down, and don't move them after seeing a result you like.
The ablation is the most informative layer. It tells you why a rule works, not just whether it did. When a rule fails layer 3, you often learn something reusable about the market — in this case, that low-volatility regimes are where this class of divergence entry lives, which is a hypothesis worth testing on its own.

More notes like this ship every week on this site.

Enforcing HTTPS on GitHub Pages After a Domain Switch

mattleeee — Sun, 25 Oct 2026 01:02:24 +0000

Pointing a new domain at GitHub Pages works on the first try often enough that the greyed-out checkbox comes as a surprise. You add the custom domain, GitHub accepts it, the site loads over plain HTTP, and the Enforce HTTPS option sits there disabled with a note that the certificate has not been provisioned yet. In most cases nothing is wrong. GitHub is still in the middle of issuing the certificate, and the toggle unlocks on its own once that finishes.

This post covers the whole lifecycle: what happens after you save a custom domain, how to watch it progress from the command line, why you should fix mixed content before you enforce HTTPS, and what to check when provisioning genuinely fails.

The provisioning sequence

When you save a custom domain for a GitHub Pages site, GitHub runs a fixed pipeline in the background. The UI gives you one checkbox as feedback, but three distinct things happen.

1. DNS check

GitHub resolves the domain and confirms it points where it should:

For an apex domain (example.com), A records must point to the Pages IPs, or an ALIAS/ANAME record must be configured.
For a subdomain (www.example.com, blog.example.com), a CNAME record must point to <user>.github.io (or <org>.github.io).

If DNS does not resolve correctly at this stage, the certificate request never starts. The UI will keep saying the certificate is pending, and it will keep saying that indefinitely. This is the failure mode worth knowing about, because it looks identical to normal waiting.

2. Certificate issuance

Once DNS checks out, GitHub requests a certificate for the domain (and the www variant where applicable) from its CA. This is the part that takes time. Typical latency is 10 to 30 minutes, but it depends on DNS TTLs and CA queueing, so an hour is not unheard of after a domain switch. During this window the site is reachable over HTTP and, in many cases, already over HTTPS with a certificate that browsers may or may not accept yet.

3. Enforce toggle unlocks

Only after the certificate is live does GitHub enable the Enforce HTTPS checkbox. Ticking it makes Pages redirect all HTTP requests to HTTPS with a 301. Until then, the box stays disabled, and there is no override, no query parameter, no API call that forces it early.

The practical takeaway: after a domain switch, expect a wait. Do not start changing DNS records while the certificate is pending, because you will reset the process.

Watching progress from the terminal

The GitHub settings page is a poor progress indicator. It shows a static message until it flips. curl tells you more.

Check what is actually being served

curl -I http://example.com
curl -I https://example.com

Run both. Before the certificate is live, the HTTPS request typically fails at the TLS handshake, which curl reports as:

curl: (60) SSL certificate problem: unable to get local issuer certificate

or a certificate mismatch error. The HTTP request, meanwhile, returns 200 with no redirect. That combination means: DNS is fine, certificate is not ready.

Once provisioning completes, https:// returns 200 and http:// still returns 200 until you tick the box. After enforcing, http:// returns:

HTTP/1.1 301 Moved Permanently
Location: https://example.com/

Inspect the served certificate

curl -I only tells you whether TLS works. To see which certificate is being served and for which names, use openssl:

echo | openssl s_client -servername example.com -connect example.com:443 2>/dev/null \
  | openssl x509 -noout -subject -issuer -dates -ext subjectAltName

A healthy result looks roughly like:

subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R3
notBefore=...
notAfter=...
X509v3 Subject Alternative Name:
    DNS:example.com, DNS:www.example.com

Two things to verify here. First, the SAN list must include every hostname you actually use. If you serve www.example.com but the certificate only covers the apex, browsers will warn on the www host even though the apex is fine. Second, check notBefore and notAfter to rule out a stale or expired certificate being served from an edge cache.

If openssl returns a certificate whose subject is *.github.io or belongs to a different domain, the request is landing on the wrong vhost. That usually means DNS is still pointing at an old host, or a proxy (Cloudflare, for example) is terminating TLS in front of Pages. Cloudflare's "Full" or "Flexible" SSL modes in front of GitHub Pages are a common source of confusing certificate output.

Scripting the wait

If you switch domains often, a small polling script saves a lot of tab-refreshing. This one checks the HTTPS handshake and the redirect status until both are correct, or gives up after a timeout:

import ssl
import socket
import time
import urllib.request
import urllib.error

DOMAIN = "example.com"
TIMEOUT_MINUTES = 45
INTERVAL_SECONDS = 120


def tls_ok(host: str, port: int = 443) -> bool:
    ctx = ssl.create_default_context()
    try:
        with socket.create_connection((host, port), timeout=10) as sock:
            with ctx.wrap_socket(sock, server_hostname=host) as tls:
                cert = tls.getpeercert()
                sans = [v for k, v in cert.get("subjectAltName", ()) if k == "DNS"]
                return host in sans or f"*.{host.split('.', 1)[1]}" in sans
    except (ssl.SSLError, socket.error, OSError):
        return False


def redirects_to_https(url: str) -> bool:
    req = urllib.request.Request(url, method="HEAD")
    try:
        with urllib.request.urlopen(req, timeout=10) as resp:
            return resp.geturl().startswith("https://")
    except urllib.error.HTTPError as e:
        return e.code in (301, 308) and e.headers.get("Location", "").startswith("https://")
    except urllib.error.URLError:
        return False


deadline = time.time() + TIMEOUT_MINUTES * 60
while time.time() < deadline:
    cert = tls_ok(DOMAIN)
    enforced = redirects_to_https(f"http://{DOMAIN}")
    print(f"cert={cert} enforced={enforced}")
    if cert and enforced:
        print("Done.")
        break
    time.sleep(INTERVAL_SECONDS)
else:
    print("Timed out. Check DNS and the Pages settings.")

The tls_ok check is stricter than a plain connection test: it validates that the hostname appears in the SAN list, which is the thing that actually matters for browsers. The redirects_to_https check confirms enforcement is active. Run it from a machine that is not behind the same DNS resolver you just changed, or you may be testing a cached view.

Fix mixed content before you enforce

This is the step people skip, and it causes the most visible breakage. Enforcing HTTPS makes the page itself load over TLS, but it does not rewrite the URLs inside your HTML, CSS, or JavaScript. Any resource still referenced as http:// becomes mixed content. Modern browsers block active mixed content (scripts, stylesheets, iframes) outright and either block or auto-upgrade passive mixed content (images), depending on the browser and the resource type.

The result: the page loads, the layout looks wrong, and the console is full of errors. Users blame the certificate; the real cause is a handful of hard-coded http:// URLs.

Find them before you flip the switch

Crawl your own site over HTTP and grep the rendered HTML for http:// references. A minimal crawler:

import re
from urllib.parse import urljoin, urlparse
from collections import deque

import requests

START = "http://example.com/"
SAME_HOST = urlparse(START).netloc

# Ignore URLs that are fine as http:// (rare) or not resources at all.
IGNORE = re.compile(r"^(mailto:|tel:|javascript:|#)")

seen = set()
queue = deque([START])
findings = []

session = requests.Session()
session.headers["User-Agent"] = "mixed-content-audit/1.0"

while queue:
    url = queue.popleft()
    if url in seen or urlparse(url).netloc != SAME_HOST:
        continue
    seen.add(url)

    try:
        resp = session.get(url, timeout=10)
    except requests.RequestException:
        continue
    if "text/html" not in resp.headers.get("Content-Type", ""):
        continue

    for match in re.finditer(r'(?:src|href|action)\s*=\s*["\'](http://[^"\']+)', resp.text):
        findings.append((url, match.group(1)))

    for match in re.finditer(r'<a\s[^>]*href\s*=\s*["\']([^"\'#]+)', resp.text):
        link = urljoin(url, match.group(1))
        if not IGNORE.match(link):
            queue.append(link)

for page, asset in findings:
    print(f"{page} -> {asset}")
print(f"\n{len(findings)} http:// references across {len(seen)} pages")

Anything it prints is a candidate for breakage. The fix is mechanical: rewrite to https://, or better, to protocol-relative or root-relative URLs (//cdn.example.com/lib.js or /assets/lib.js) so the scheme follows the page.

Common offenders

Hard-coded CDN links in templates, often copied from documentation years ago.
Markdown content with embedded images using http:// URLs.
Analytics or font snippets pasted from a provider dashboard that predates HTTPS-by-default.
srcset and <picture> sources, which are easy to miss because they are not plain src attributes.
CSS url() references to background images or fonts.

The crawler above only covers HTML attributes. For a thorough pass, also fetch your CSS files and search them for http://, and check any client-side JavaScript that builds URLs at runtime. A Content-Security-Policy header with upgrade-insecure-requests is a useful safety net, but it is a mitigation, not a fix. Do the rewrite first.

When provisioning actually fails

If more than a couple of hours pass and curl -I https://example.com still fails the handshake, the pipeline is stuck. Work through these in order.

Verify the CNAME value exactly

For a subdomain, the CNAME record must point to the Pages hostname, not to an IP, not to a proxy, and not to a URL with a path. The value is <user>.github.io. (or <org>.github.io.), with or without the trailing dot depending on your DNS provider. The apex domain is a special case: it cannot be a CNAME, so it needs A records to the documented GitHub Pages IPs, plus a CNAME file in the repository or the custom domain set in settings.

dig +short CNAME www.example.com
dig +short A example.com
dig +short example.com NS

Check the NS output too. If the domain is still delegated to your registrar's parking nameservers because you changed nameservers recently, the records you configured in the new DNS provider are not live yet. That alone will stall provisioning.

Wait out the TTL

If you changed DNS records within the last 48 hours, resolvers across the internet are still serving cached answers with the old target. GitHub's checker may have queried a resolver that still had the old record and concluded DNS was wrong. There is no way to force GitHub's check; it retries on its own schedule. Lowering the TTL before a planned migration (to 300 seconds, a day or two in advance) makes this window much shorter.

To see what different resolvers think, query a public resolver directly:

dig +short CNAME www.example.com @1.1.1.1
dig +short CNAME www.example.com @8.8.8.8

If these disagree, you are still inside the propagation window. Wait.

Remove and re-add the custom domain

If DNS is correct across resolvers, the served certificate is wrong or absent, and several hours have passed, remove the custom domain in the Pages settings, save, and add it back. This cancels the stuck request and starts a fresh one. It is safe, but it resets the clock, so only do it once you are confident DNS is right. Re-adding the domain while DNS is still wrong just reproduces the same stuck state.

Rule out a proxy in front of Pages

If the domain is behind Cloudflare or a similar proxy, the certificate GitHub issues may never be served, because the proxy terminates TLS with its own certificate. That is not necessarily a problem, but it changes where you look: the openssl output will show the proxy's issuer, and the "Enforce HTTPS" toggle in GitHub may stay disabled because GitHub cannot complete its own validation through the proxy. Either set the proxy to DNS-only during provisioning and re-enable it afterwards, or accept that HTTPS is handled at the proxy layer and configure enforcement there instead.

Check the CNAME file in the repo

If you set the custom domain through the repository's CNAME file rather than the settings UI, a mismatch between the file contents and the settings value can cause the domain to flap between configured states. Pick one method and stick to it; the settings UI writes the CNAME file for you.

A short checklist

Before enforcing HTTPS after a domain switch:

Confirm DNS resolves correctly from multiple public resolvers.
Wait for the certificate; verify it with openssl s_client and check the SAN list covers every hostname.
Crawl the site for http:// asset references and rewrite them.
Tick Enforce HTTPS once the checkbox unlocks.
Verify the redirect with curl -I http://example.com and re-run the crawler against the HTTPS origin to confirm nothing regressed.

The greyed-out checkbox is normal. The failures that matter are the ones where it stays greyed out for hours, and those almost always trace back to DNS.

More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-25 08:56:33

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
AXIS	短平快做多	$0.2484	$0.2559	$0.2422	1:1.2
ORDER	短平快做多	$0.0566	$0.0583	$0.0552	1:1.2

2 picks selected. Scanner runs every 15 minutes.

Why You Cannot Put a CNAME on the Apex Domain

mattleeee — Sat, 24 Oct 2026 01:01:45 +0000

The zone apex is not like other names

The apex of a zone — the bare domain itself, hkcode.dpdns.org in this case — is where the DNS machinery for the entire zone lives. It must carry an SOA record and a set of NS records. That single fact is the reason you cannot point the apex at a CNAME, and it is not a limitation of GitHub Pages or of any registrar. It is baked into the DNS standard.

When we set up GitHub Pages for a custom domain, GitHub's documentation asks for a CNAME record. The natural reading is "point the domain at username.github.io." For the apex, that request cannot be satisfied literally. The site ends up at www.hkcode.dpdns.org, and the bare name is handled by a redirect or a set of A/AAAA records. Here is why, and how to wire it up without creating a resolver loop or a broken zone.

Why a CNAME cannot live at the apex

The problem: a CNAME claims the whole name

RFC 1034, section 3.6.2 is explicit: a CNAME record is an alias for a domain name, and that name must not have any other data. The text says that if a CNAME RR is present at a node, no other data should be present. RFC 2181, section 10.1 reinforces it: a CNAME record cannot coexist with any other record type at the same owner name, with a narrow exception for DNSSEC-related records (RRSIG, NSEC, and similar).

So the rule is: at a node, either a CNAME, or everything else. Not both.

Now look at the apex. Every zone needs:

An SOA record, which describes the zone's primary server, serial number, refresh/retry/expire timers, and negative-caching TTL.
At least one NS record set, listing the authoritative name servers.

Those records must exist at the apex by definition — the SOA names the zone origin, and the NS set tells the world who is authoritative for it. If you also tried to place a CNAME at the apex, you would be asking the zone to hold a CNAME and an SOA and NS records at the same owner name. That directly violates RFC 1034/2181.

What happens if a provider lets you try

Many authoritative servers will simply reject the zone file at load time with an error like CNAME and other data. Some permissive servers accept it, and the result is worse: resolvers receive an answer where the CNAME and the SOA/NS coexist. Behavior becomes undefined. Some resolvers will follow the CNAME and discard the SOA; some will treat the zone as lame. Recursive resolvers that cache the CNAME may return it for queries that should have hit the NS set. You end up with intermittent resolution failures, and the failure mode differs by resolver — the hardest kind of bug to diagnose.

There is also a subtler issue. A CNAME at the apex would alias the entire zone's identity to another name. Queries for the SOA at the apex would be answered with a CNAME, and the resolver would chase it to the target, which has no authority over your zone. The delegation chain breaks.

The MX and TXT trap

Even if the RFC allowed it, the apex usually carries more than SOA and NS. Email routing records (MX) and domain-verification or SPF/DKIM records (TXT) commonly live at the bare domain. A CNAME at the apex would make all of those illegal too. For hkcode.dpdns.org, any TXT record used to verify ownership with a third-party service would collide with the CNAME. This is why the "just CNAME the apex" idea fails beyond the pure DNS mechanics: real zones have real records at the apex.

What "flattening" providers actually do

Several DNS providers (Cloudflare, DNSimple, Route 53 with alias records, and others) offer a record type called ALIAS, ANAME, or a "CNAME flattening" feature. This is the workaround people reach for, but it is not a CNAME at the apex. It is a server-side behavior.

Here is the mechanism. You create an ALIAS/ANAME record at the apex pointing to a target name. The provider's authoritative servers do not publish a CNAME. Instead, when a query for the apex arrives, the authoritative server:

Resolves the target name itself (or uses its cached answer).
Returns the resulting A/AAAA records directly, as if they were A/AAAA records at the apex.

The zone on disk still has SOA and NS at the apex; the ALIAS is a provider-specific synthetic record. To the outside world, the apex answers with address records, not a CNAME, so the RFC constraint is respected. The provider has effectively flattened the indirection.

This works, but it is not portable. The ALIAS record type is not standardized in the way A and CNAME are; its semantics differ between providers. If you export the zone and move to a provider that does not support it, the record either disappears or is written out as a literal CNAME, which then breaks the zone. Also, the target's TTL becomes the provider's resolution TTL, not the original CNAME TTL, so you lose some control over caching.

For GitHub Pages specifically, the target username.github.io resolves to GitHub's CDN addresses. Flattening works, but you are pinning the apex to whatever those addresses are at query time. If GitHub rotates them, your provider's cached resolution handles it, but you are trusting the provider's resolver behavior.

How GitHub Pages handles www versus apex

GitHub Pages has a specific model for custom domains:

www subdomain: GitHub recommends a CNAME record pointing www.hkcode.dpdns.org to hkcode.github.io (or username.github.io). This is the canonical setup. GitHub's servers see the Host: www.hkcode.dpdns.org header, match it against the configured custom domain, and serve the site.
Apex domain: GitHub documents A records (and AAAA records for IPv6) pointing the apex at GitHub's published IP addresses. These are static addresses that GitHub maintains. The apex is not a CNAME; it is a set of address records.

When you configure a custom domain in the repository settings, GitHub stores it and serves requests whose Host header matches. If you set the custom domain to www.hkcode.dpdns.org, requests to the apex hkcode.dpdns.org will not match unless GitHub also handles the redirect.

GitHub Pages does provide an HTTP-level redirect from the apex to the www host when the custom domain is configured as the www name and the apex resolves to GitHub. The redirect is a 301 served by GitHub's front end, not a DNS-level redirect. For that to work, the apex must still resolve to GitHub's servers — via A/AAAA records — so the request reaches GitHub and gets redirected. If the apex has no records at all, the browser never reaches GitHub and there is no redirect.

So the practical split is:

DNS for www → CNAME to username.github.io.
DNS for apex → A/AAAA records to GitHub's addresses (or a provider redirect).
GitHub config → custom domain set to the www name; GitHub 301s the bare name to www.

The practical setup

Step 1: CNAME the www subdomain

At your DNS provider, create:

www.hkcode.dpdns.org.  3600  IN  CNAME  hkcode.github.io.

No other records at www. If you previously had an A record there, remove it. A CNAME and an A record at the same name is the same violation as at the apex, just at a different node.

Step 2: Handle the apex

Two options, depending on your provider.

Option A — GitHub's A/AAAA records. Create A records (and AAAA if you want IPv6) at the apex pointing to GitHub's published addresses. These are documented by GitHub and change rarely, but check them before you paste. The apex then answers with addresses, and GitHub's front end 301s to www.

Option B — provider redirect. If your provider offers an HTTP redirect service for the apex (some registrars and DNS hosts do), point the apex at that redirect and send it to https://www.hkcode.dpdns.org. This avoids hardcoding GitHub's IPs. The tradeoff is that the redirect is handled by the provider, so you depend on their uptime and their support for HTTPS on the redirect.

Do not use an ALIAS/ANAME at the apex unless you have verified the provider's implementation and are comfortable with the non-portability. It works, but it is a provider feature, not a DNS standard.

Step 3: Verify with code

Do not trust the control panel. Query the records directly and check the invariants. A small script using dnspython catches the common mistakes: a stray CNAME at the apex, a missing SOA, or a CNAME and A record coexisting at www.

import dns.resolver
import dns.exception

APEX = "hkcode.dpdns.org"
WWW = f"www.{APEX}"

def query(name, rtype):
    try:
        answer = dns.resolver.resolve(name, rtype, lifetime=5)
        return [r.to_text() for r in answer]
    except dns.resolver.NoAnswer:
        return []
    except dns.resolver.NXDOMAIN:
        return ["NXDOMAIN"]
    except dns.exception.Timeout:
        return ["TIMEOUT"]

def check_zone(apex, www):
    problems = []

    # The apex must have SOA and NS. A CNAME here is illegal.
    apex_cname = query(apex, "CNAME")
    if apex_cname and apex_cname != ["NXDOMAIN"]:
        problems.append(f"apex has a CNAME: {apex_cname}")

    if not query(apex, "SOA"):
        problems.append("apex has no SOA")
    if not query(apex, "NS"):
        problems.append("apex has no NS")

    # The www node should be a CNAME and nothing else.
    www_cname = query(www, "CNAME")
    www_a = query(www, "A")
    if not www_cname:
        problems.append("www has no CNAME")
    if www_a and www_cname:
        problems.append("www has both CNAME and A records")

    return problems

if __name__ == "__main__":
    issues = check_zone(APEX, WWW)
    if issues:
        for issue in issues:
            print("FAIL:", issue)
    else:
        print("zone looks consistent")

Run it against the authoritative servers, not just your local resolver, so you see what the world sees. You can set the resolver explicitly:

import dns.resolver

resolver = dns.resolver.Resolver(configure=False)
# Use the provider's authoritative nameservers, from the NS set.
resolver.nameservers = ["ns1.example.net", "ns2.example.net"]
resolver.lifetime = 5

Step 4: Check the HTTP redirect

Once DNS is correct, confirm the redirect behavior with a request that does not follow redirects:

import urllib.request
import urllib.error

def check_redirect(url):
    req = urllib.request.Request(url, method="HEAD")
    opener = urllib.request.build_opener(urllib.request.HTTPRedirectHandler)
    try:
        # Prevent automatic following so we can inspect the 301.
        class NoRedirect(urllib.request.HTTPRedirectHandler):
            def redirect_request(self, *args, **kwargs):
                return None
        opener = urllib.request.build_opener(NoRedirect)
        resp = opener.open(req, timeout=10)
        print(url, "->", resp.status)
    except urllib.error.HTTPError as e:
        location = e.headers.get("Location")
        print(url, "->", e.code, "Location:", location)
    except Exception as e:
        print(url, "-> error:", e)

check_redirect("http://hkcode.dpdns.org/")
check_redirect("https://hkcode.dpdns.org/")

You want a 301 (or 308) with Location: https://www.hkcode.dpdns.org/. If you get a connection error, the apex has no address records and the redirect never happens. If you get a 200, GitHub is serving the apex directly — check whether the custom domain in the repo settings is set to the bare name, which changes the canonical host.

Common failure modes

CNAME at the apex "because the provider allowed it." The zone loads, but some resolvers fail on the SOA or NS query. Symptom: intermittent SERVFAIL from specific networks.
CNAME at www plus an A record "for safety." Same violation, smaller blast radius. Remove the A record.
Apex A records but no www CNAME. The site works at the bare name but not at www, or vice versa, and the canonical URL drifts.
Provider redirect without HTTPS. The apex redirects over HTTP only, and browsers block or warn. Make sure the redirect target uses https://.
Stale TTLs during migration. Lower the TTL on the old records to 300 seconds a day before the change, then raise it after the switch settles.

Summary

A CNAME at the apex is not a configuration choice; it is a contradiction of the zone's required SOA and NS records under RFC 1034 and 2181. ALIAS/ANAME records are a provider-side flattening trick that returns address records instead of a CNAME, and they work but do not travel. GitHub Pages expects a CNAME on www and either A/AAAA records or a provider redirect on the apex, with an HTTP 301 from the bare name to www. Set the CNAME on the subdomain, give the apex address records or a redirect, and verify both the DNS invariants and the redirect with a script rather than the control panel.

More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-23 08:46:52

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
BTW	高质量标的做多	$1.8281	$1.9561	$1.7641	1:2.0

1 picks selected. Scanner runs every 15 minutes.

Serving a Bilingual Static Site on GitHub Pages

mattleeee — Sat, 24 Oct 2026 01:01:42 +0000

Why one repo, two sites

A Chinese blog already lived at the domain root: example.com/. The goal was an English sibling at example.com/en/ without a second repository, a second domain, or a second deploy pipeline. That constraint drives every decision below.

The layout that ended up working:

repo/
├── CNAME                 # single domain for both languages
├── published/            # Chinese article manifests (one JSON per post)
├── published_en/         # English article manifests
├── templates/            # shared design system, Jinja2
├── static/               # shared CSS, JS, fonts
├── build.py              # generator for the root site
├── build_en.py           # generator for /en
└── docs/                 # GitHub Pages output (root = Chinese, /en = English)
    ├── index.html
    ├── sitemap.xml
    ├── rss.xml
    └── en/
        ├── index.html
        ├── sitemap.xml
        └── rss.xml

GitHub Pages serves docs/ from the branch, so the directory tree is the URL tree. The Chinese site is the root; the English site is a subdirectory. One CNAME file, one certificate, one git push.

Separate manifests prevent cross-language idempotency bugs

The first version kept one manifest directory with a lang field per article. Generation was supposed to be idempotent: run build.py twice, get identical output. It wasn't. The bug came from how the generator picked slugs and resolved "previous/next" links.

With a single pool, the Chinese generator saw English articles as candidates for adjacency and for slug-collision checks. A Chinese post at 2024/03/foo and an English post at 2024/03/foo (same slug, different language) would race for the same output path. Whichever ran second silently overwrote the first. Worse, rerunning the build could change which one won if the manifest glob order shifted. Non-deterministic output from deterministic input is the worst class of build bug: it looks fine locally, then a post disappears in production a week later.

The fix is boring and total: two manifest directories, published/ and published_en/, and each generator reads only its own. No lang filter, no shared slug namespace, no cross-language adjacency. The English generator cannot see Chinese manifests, so it cannot be affected by them.

from pathlib import Path
import json

MANIFEST_DIRS = {
    "zh": Path("published"),
    "en": Path("published_en"),
}

def load_manifests(lang: str) -> list[dict]:
    """Load only this language's manifests, sorted newest first."""
    posts = []
    for path in MANIFEST_DIRS[lang].glob("*.json"):
        with path.open(encoding="utf-8") as f:
            post = json.load(f)
        post["_source"] = path.name
        posts.append(post)
    posts.sort(key=lambda p: p["date"], reverse=True)

    slugs = [p["slug"] for p in posts]
    if len(slugs) != len(set(slugs)):
        raise ValueError(f"duplicate slug in {lang} manifests")

    return posts

The duplicate-slug check now guards a single language, which is exactly the invariant you want: slugs must be unique within a site, not across sites, because the two sites live at different URL prefixes.

The generators share the design system but nothing else. build.py and build_en.py both import templates/ and static/, both write into docs/, but each owns its manifest set and its own URL prefix. If the English site needs a different layout tweak later, it can diverge without touching the Chinese build.

One prefix constant keeps links consistent

Every internal link needs two pieces: the site root ("" for Chinese, "en" for English) and the article path. Hardcoding either one is how you end up with an English page linking to a Chinese article, or a link that works locally (/en/...) but 404s under a project-page base path.

The fix is a single constant per build plus a helper that joins them. Nothing else in the codebase constructs internal URLs by string concatenation.

# In build.py
SITE_PREFIX = ""          # Chinese site at the domain root
LANG = "zh"

# In build_en.py
SITE_PREFIX = "en"        # English site under /en
LANG = "en"

Both files import one shared module for URL construction:

# urls.py
def site_url(prefix: str, path: str = "") -> str:
    """Absolute-path URL for an internal page.

    site_url("", "about/")      -> "/about/"
    site_url("en", "about/")    -> "/en/about/"
    """
    parts = [p.strip("/") for p in (prefix, path) if p.strip("/")]
    return "/" + "/".join(parts) + ("/" if path.endswith("/") or not path else "")

Then the templates never see raw paths. The build passes a url function into the Jinja2 environment:

from jinja2 import Environment, FileSystemLoader
from urls import site_url

def make_env(prefix: str) -> Environment:
    env = Environment(loader=FileSystemLoader("templates"), autoescape=True)
    env.globals["url"] = lambda path="": site_url(prefix, path)
    return env

In templates, links become {{ url('posts/' + post.slug + '/') }} and assets become {{ url('static/main.css') }}. Because url closes over the prefix, the same template renders correct links for both languages. A Chinese page emits /posts/foo/; an English page emits /en/posts/foo/. There is no branch in the template and no opportunity to forget the prefix.

The asset trap

Shared assets under static/ are the one place where a prefix-relative link is wrong. If the English page links to /en/static/main.css but the file only exists at /static/main.css, you get a 404 on every English page. Two options: emit absolute asset URLs from the domain root, or mirror the assets under both prefixes. Mirroring doubles the files and the cache entries. The cleaner choice is a separate helper for assets that ignores the language prefix:

env.globals["asset"] = lambda path: "/static/" + path.lstrip("/")

Templates use {{ asset('main.css') }} for CSS/JS/fonts and {{ url(...) }} for anything that belongs to a language. One rule, applied consistently, and the asset 404s stop.

Cross-language navigation in both navs

Each nav needs a link to the equivalent page in the other language, not just the other homepage. On an article page, "English" should go to that article's English version if it exists; otherwise fall back to the English index.

That requires a mapping between the two manifest sets. The simplest reliable key is an explicit translation_key field present in both manifests for paired articles:

def build_translation_map(zh_posts, en_posts):
    """Map (lang, translation_key) -> slug for cross-language links."""
    mapping = {}
    for lang, posts in (("zh", zh_posts), ("en", en_posts)):
        for post in posts:
            key = post.get("translation_key")
            if key:
                mapping[(lang, key)] = post["slug"]
    return mapping

When rendering a Chinese post with translation_key: "gh-pages-bilingual", the nav looks up ("en", "gh-pages-bilingual"). If found, the link points to /en/posts/<slug>/; if not, it points to /en/. Same logic in reverse for the English site. Articles without a translation still get a working language switch, just at the section root.

def alt_lang_href(post, mapping, target_lang, target_prefix):
    key = post.get("translation_key")
    slug = mapping.get((target_lang, key)) if key else None
    if slug:
        return site_url(target_prefix, f"posts/{slug}/")
    return site_url(target_prefix)

The hreflang tags in <head> use the same function, so the machine-readable and human-readable links can't drift apart:

def hreflang_links(post, mapping):
    links = [("zh", site_url("", f"posts/{post['slug']}/") if post["lang"] == "zh"
              else site_url("", f"posts/{mapping[('zh', post['translation_key'])]}/"))]
    # ... same for en, plus x-default pointing at the Chinese page

In practice it's cleaner to compute both URLs once and pass a small dict into the template, but the principle holds: cross-language links are derived from the translation map, never typed by hand.

Per-language sitemaps and feeds

A single root sitemap.xml listing both languages is legal, but it makes Search Console reporting muddy and forces every URL to carry hreflang annotations in one file. Since the two sites already have separate manifests and separate generators, each generator writes its own sitemap and feed under its own prefix:

docs/sitemap.xml and docs/rss.xml — Chinese
docs/en/sitemap.xml and docs/en/rss.xml — English

from xml.sax.saxutils import escape
from datetime import datetime

def write_sitemap(posts, prefix: str, out_path: Path, base: str = "https://example.com"):
    urls = [f"{base}{site_url(prefix)}"]
    urls += [f"{base}{site_url(prefix, f'posts/{p[\"slug\"]}/')}" for p in posts]

    body = "\n".join(
        f"  <url><loc>{escape(u)}</loc></url>" for u in urls
    )
    xml = (
        '<?xml version="1.0" encoding="UTF-8"?>\n'
        '<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n'
        f"{body}\n</urlset>\n"
    )
    out_path.write_text(xml, encoding="utf-8")

The RSS feed follows the same shape, reading <link> values from site_url(prefix, ...). Because both use the same prefix constant, a feed can never contain a URL from the wrong language.

If you want the sitemaps discoverable without submitting each one manually, add a sitemap index at the root:

def write_sitemap_index(base: str, out_path: Path):
    entries = [
        f"{base}/sitemap.xml",
        f"{base}/en/sitemap.xml",
    ]
    body = "\n".join(
        f"  <sitemap><loc>{escape(u)}</loc></sitemap>" for u in entries
    )
    out_path.write_text(
        '<?xml version="1.0" encoding="UTF-8"?>\n'
        '<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">\n'
        f"{body}\n</sitemapindex>\n",
        encoding="utf-8",
    )

After the domain switch: submit the second sitemap

The domain move is where this layout earns its keep. When the site moved to a custom domain, the CNAME file changed once and both languages moved together. GitHub Pages issued the certificate for the new domain, and the root site's URLs redirected cleanly because they were always relative to the domain root.

The English site needed one extra step. Search Console properties are per-domain (or per-URL-prefix), and the new domain was verified fresh. The root sitemap.xml was submitted immediately, but /en/sitemap.xml sat unsubmitted for a week. English pages were indexed slowly and with stale titles. The fix was to open the new property's Sitemaps report and add https://example.com/en/sitemap.xml as a second sitemap. Within days the English pages were crawled at the same rate as the Chinese ones.

Two practical notes from that episode. First, submit the sitemap index if you wrote one; Search Console will follow it. Second, the hreflang annotations matter more after a domain change, because the crawler is re-evaluating which URLs are alternates of which. If your language switch links were broken by the move, the annotations would point at dead URLs and the English pages would be treated as orphans. Keeping the language switch derived from the translation map, as above, means a domain change doesn't touch link logic at all.

What the layout buys you

The whole design rests on three invariants: one manifest directory per language, one prefix constant per build, one URL helper used everywhere. Everything else — sitemaps, feeds, hreflang, cross-language nav — falls out of those. The build is idempotent because the two generators can't see each other's inputs. Links are consistent because no template constructs a URL by hand. And adding a third language later is a matter of a new manifest directory, a new prefix, and a new generator file, not a refactor.

More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-24 08:47:12

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
BIRB	弱做多	$0.1706	$0.1835	$0.1621	1:1.5

1 picks selected. Scanner runs every 15 minutes.

The CNAME Trailing Dot: One Character That Broke the Domain

mattleeee — Thu, 22 Oct 2026 01:01:29 +0000

The CNAME Trailing Dot: One Character That Broke the Domain

The migration was supposed to take ten minutes. New domain registered on a free-tier registrar, GitHub Pages already serving the site on mattlee.github.io, so all that remained was a CNAME record pointing www.example.dev at the GitHub Pages hostname. I typed mattlee.github.io into the panel's value field, saved, and waited. Then I waited some more.

dig returned NXDOMAIN for the apex and SERVFAIL for www. GitHub Pages reported no custom domain configured. The DNS panel insisted the record existed, and it did — just not the record anyone intended to create.

The fix was a single character. The value needed to be mattlee.github.io., with the trailing dot. Without it, the registrar stored a relative name, and the record resolved to mattlee.github.io.example.dev. — a hostname inside my own zone that points at nothing.

What the trailing dot actually means

DNS has two kinds of names: absolute (fully qualified) and relative. The distinction is encoded in the presentation format by a terminal dot. mattlee.github.io. is an absolute name — it is fully specified and the resolver stops there. mattlee.github.io is a relative name — it is a prefix to be completed by appending an origin.

This is not a quirk of one vendor. RFC 1034 and RFC 1035 define the master file format with this convention: a domain name that does not end in a dot is interpreted relative to the current origin. In a zone file, the origin is the zone itself. So inside the zone for example.dev, the relative name mattlee.github.io expands to mattlee.github.io.example.dev.

A zone file makes this visible:

$ORIGIN example.dev.
www     IN  CNAME  mattlee.github.io.      ; absolute: mattlee.github.io.
blog    IN  CNAME  mattlee.github.io       ; relative: mattlee.github.io.example.dev.

Both lines look plausible in a web form. Only the first one does what the author meant. The second creates a CNAME that points back into the same zone, to a name that has no A record. Resolvers chase it, find nothing, and return NXDOMAIN or SERVFAIL depending on how far the chain gets.

Why does the protocol work this way at all? Because relative names are useful. Zone files are full of them — www, mail, api — and appending the origin keeps configuration short and portable across zones. The trailing dot is the escape hatch that says "I am done, this is complete." The format is terse by design, and the dot is the only signal that distinguishes the two cases.

Why some panels add the dot and some don't

Registrar and DNS-provider UIs sit on a spectrum between two philosophies.

Zone-file faithful. Some panels treat the value field exactly as a zone file would. If you omit the dot, you get a relative name. These panels are technically correct and operationally hostile — most users typing a CNAME value mean an absolute hostname, because they copied it from a documentation page or a hosting dashboard where the trailing dot is usually omitted.

Lenient normalizers. Other panels assume every value is absolute and append the dot for you on save. Many also strip a dot you type in, then re-add it, so the stored record is always fully qualified. These panels hide the problem entirely, which is why the same record that works on one provider fails on another.

Hybrid. A third class stores what you type but normalizes on display, so the panel shows mattlee.github.io whether or not the dot is present. You cannot tell from the UI what was actually written to the zone. This is the worst case for debugging, because the panel's appearance actively misleads you.

The practical consequence: a CNAME value that works on your old registrar may break on your new one, even though both panels display the same string. The record is not the string you see. The record is what the provider serialized into the zone.

How to verify what was actually stored

Never trust the panel. Query the authoritative nameservers directly and read the ANSWER section.

dig www.example.dev CNAME +short

If the record is correct, you get:

mattlee.github.io.

Note the trailing dot in the output. dig always prints absolute names with the dot, because that is what the wire format contains — DNS messages carry fully qualified names, and the presentation layer adds the dot back on output.

If the record is broken, you get something like:

mattlee.github.io.example.dev.

That is the smoking gun. The resolver is telling you the stored name is relative and the origin was appended. If you see the zone name glued onto the end of your intended target, the value field is missing its dot.

You can also query the authoritative server directly, bypassing caching and any recursive resolver behavior:

dig @ns1.registrar.net www.example.dev CNAME +noall +answer

The +noall +answer flags trim the output to just the answer section, which is where the CNAME target lives. If the answer section is empty and the authority section shows NXDOMAIN, the name may not exist at all — check that the record saved and that the zone is active.

nslookup works too, though its output is noisier:

nslookup -type=CNAME www.example.dev ns1.registrar.net

Look at the canonical name = line. Same rule: a trailing dot means absolute, no trailing dot in a zone-file-style answer means relative.

For a full trace of the resolution chain, including where it fails:

dig www.example.dev +trace

+trace walks from the root down. If the delegation to your zone succeeds but the CNAME target never resolves, the failure is inside your zone — almost always the missing dot.

Automating the check

Once you know what to look for, the check is trivial to script. This is worth wiring into your deploy pipeline, because a broken CNAME is invisible until someone tries to load the site.

import subprocess
import sys


def get_cname(name: str, server: str | None = None) -> str | None:
    """Return the CNAME target for `name`, or None if no CNAME exists."""
    cmd = ["dig", "+short", "CNAME", name]
    if server:
        cmd.insert(1, f"@{server}")
    result = subprocess.run(cmd, capture_output=True, text=True, timeout=10)
    target = result.stdout.strip()
    return target or None


def check_cname(name: str, expected: str, server: str | None = None) -> bool:
    """
    Verify that `name` is a CNAME to `expected`.

    `expected` should be given without a trailing dot; the comparison
    normalizes both sides so the check is independent of presentation.
    """
    target = get_cname(name, server)
    if target is None:
        print(f"FAIL: no CNAME record for {name}")
        return False

    normalized = target.rstrip(".")
    if normalized != expected.rstrip("."):
        print(f"FAIL: {name} -> {target} (expected {expected}.)")
        print("      If the target ends in your zone name, the value is")
        print("      relative and is missing its trailing dot.")
        return False

    print(f"OK: {name} -> {target}")
    return True


if __name__ == "__main__":
    ok = check_cname("www.example.dev", "mattlee.github.io")
    sys.exit(0 if ok else 1)

The rstrip(".") normalization matters: dig +short prints the absolute form with a dot, but you will often pass the expected value without one, as copied from a dashboard. Comparing the normalized forms avoids false alarms while still catching the real failure — a target that has your zone name appended.

If you want to detect the relative-name bug specifically, check whether the target ends with the zone:

def looks_relative(target: str, zone: str) -> bool:
    """Heuristic: a CNAME target that ends in the zone is probably relative."""
    return target.rstrip(".").endswith(zone.rstrip("."))

This is a heuristic, not a proof — a legitimate CNAME can point into your own zone, and internal service records often do. But for a public web CNAME pointing at an external host, a target that ends in your own zone name is a near-certain sign of a missing dot.

The general lesson

DNS tools are literal and unapologetic. There is no fuzzy matching, no "did you mean," no warning when you save a relative name where you meant an absolute one. The zone is a text format with a precise grammar, and the provider either follows it or silently normalizes it for you. When it follows it, you get exactly what you typed, including the consequences.

This bites in predictable places:

CNAME and MX values are the most common victims, because they are the record types where you type a hostname rather than an IP address.
Copy-paste from documentation usually drops the trailing dot, because docs are written for humans and the dot is noise to most readers.
Provider migration surfaces latent bugs, because the new panel may interpret the value differently from the old one.
TXT records are not affected the same way — the content is an opaque string, not a domain name — but SPF and DKIM records that embed domain names inside the string can carry the same confusion.

The habit that prevents all of it: when a record value is a hostname, type the trailing dot. It is harmless on panels that normalize, and load-bearing on panels that don't. Then verify with dig and read the ANSWER section, because the panel is a UI and the zone is the truth.

The domain came up the moment the dot was added. The record had been "correct" in the panel for an hour. It had never been correct in the zone.

More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-22 08:46:44

Short

Pair	Signal	Price	Take Profit	Stop Loss	R/R
ARIA	短平快做空	$0.0655	$0.0635	$0.0671	1:1.2

1 picks selected. Scanner runs every 15 minutes.

Free Domain Options in 2026: From eu.org to DigitalPlat dpdns.org

mattleeee — Wed, 21 Oct 2026 01:01:48 +0000

The 2026 Free Domain Landscape

If you run static sites for side projects, a domain is the one recurring cost you can usually avoid. The catch is that free domain providers are unstable in a way paid registries are not. Programs open, close, change terms, or quietly stop approving applications. Planning around them means treating the domain as a replaceable pointer, not as permanent infrastructure.

The 2026 situation, based on what actually happened over the past couple of years:

eu.org still works and is still free, but manual review can take months. It is fine for a project you are just starting; it is not a tool for a site that needs to be live this week.
DigitalPlat's .us.kg and .xx.kg paused new registrations after abuse problems. Existing domains kept resolving, but you could not onboard.
.qzz.io, also run under the DigitalPlat umbrella, moved to paid-only.
dpdns.org stayed open for new registrations and became the default choice for people who needed a free subdomain quickly.

The pattern is obvious: subdomain programs are cheap to launch and expensive to police. Expect any free suffix to have a finite lifespan. The engineering response is to make migration cheap, which is what most of this post is about.

Why dpdns.org Became the Default

dpdns.org is a subdomain service, not a registrar. You get something like yourname.dpdns.org, and the provider runs the authoritative nameservers for the zone. You control the records beneath your label.

What makes it usable for static hosting:

New registrations are open, so you can get a name in minutes rather than months.
You can delegate to external nameservers if you want, which matters if you already run DNS elsewhere.
It works with GitHub Pages, Cloudflare Pages, Netlify, and Vercel without special handling, because those platforms only care about a CNAME pointing at their edge.
HTTPS works through the hosting platform's certificate automation, so you do not need to manage TLS for the subdomain yourself.

The trade-offs are the same as any free subdomain. You do not own the name. The zone owner can change rules. There is no EPP transfer, so you cannot move the domain to another registrar—only the records inside it. And whois is essentially meaningless: the registrant is the provider, and your identity lives only in their control panel.

That is acceptable for a portfolio, a documentation mirror, or a demo site. It is not acceptable for anything where losing the hostname breaks users, email, or payments.

The DigitalPlat Registration Flow

The flow is short, but the DNS step is where people get stuck.

1. Register the label

Sign in to the DigitalPlat panel, pick the suffix (dpdns.org), and enter the label you want. Registration is effectively first-come-first-served, so short labels go fast. Once accepted, the subdomain appears in your dashboard.

At this point the domain exists but resolves to nothing useful. You have to point it somewhere.

2. Choose between provider DNS and external nameservers

Two paths:

Use DigitalPlat's nameservers. You add A, AAAA, or CNAME records in their panel. Simplest option, and fine for a single static site.
Delegate to external nameservers. You set NS records for your subdomain to, say, Cloudflare's nameservers, then manage everything there. More control, and it makes the next migration easier because your records live in a zone you can export.

For static sites, the second path is worth the extra ten minutes. Keeping records in a zone you control means that when dpdns.org changes terms, you only need to repoint the NS delegation—your A records, CNAMEs, and TXT records stay put.

3. Point the records at your host

For GitHub Pages, the pattern is a CNAME from your subdomain to <user>.github.io, plus the custom domain set in the repository settings. For Cloudflare Pages or Netlify, it is a CNAME to the project's assigned hostname. Either way, the record is a single CNAME and the platform handles the certificate.

If you are delegating to external nameservers, you create that CNAME in the external zone, not in the DigitalPlat panel. Doing it in both places causes conflicting answers and confusing propagation behavior.

4. Verify resolution before you announce anything

Do not trust the panel's "active" status. Query the authoritative servers directly:

import dns.resolver

def check_subdomain(name: str, record_type: str = "CNAME") -> None:
    """Query the authoritative chain for a subdomain and print what it returns."""
    resolver = dns.resolver.Resolver(configure=False)
    resolver.nameservers = ["1.1.1.1", "8.8.8.8"]
    resolver.lifetime = 5.0

    try:
        answers = resolver.resolve(name, record_type)
    except dns.resolver.NoAnswer:
        print(f"{name} {record_type}: no answer")
        return
    except dns.resolver.NXDOMAIN:
        print(f"{name}: NXDOMAIN")
        return

    for rdata in answers:
        print(f"{name} {record_type} -> {rdata}")

check_subdomain("yourname.dpdns.org", "CNAME")

If you delegated NS records, also confirm the delegation itself took effect:

import dns.resolver

def check_delegation(name: str) -> None:
    """Print the NS records that the parent zone actually serves for a subdomain."""
    resolver = dns.resolver.Resolver(configure=False)
    resolver.nameservers = ["1.1.1.1"]
    try:
        for ns in resolver.resolve(name, "NS"):
            print(f"NS: {ns.target}")
    except dns.resolver.NoAnswer:
        print(f"{name}: no NS records served by parent")

check_delegation("yourname.dpdns.org")

If the NS query returns nothing, the delegation has not propagated or was never saved. Fix that before touching anything downstream—a CNAME in a zone nobody is querying will never resolve.

What You Give Up

Be explicit about the costs, because they shape how you should use these domains.

Stability risk. The zone owner can change policy at any time. .qzz.io going paid-only is the concrete example; .us.kg pausing registrations is another. Your domain is not a contract, it is a courtesy.

No whois control. The registrant record belongs to the provider. You cannot prove ownership through whois, which rules out some verification workflows and makes disputes unwinnable.

No transfer. There is no EPP code, so you cannot move the name to another registrar. Migration means registering a new name and repointing traffic.

Certificate and email constraints. Let's Encrypt issuance works fine for the web hostname, but you should not build email on a free subdomain. SPF, DKIM, and DMARC on a name you do not own is a deliverability problem waiting to happen.

Search and reputation. Shared suffixes carry the reputation of everyone else on them. A subdomain on a suffix with heavy abuse gets treated worse by filters and some mail systems.

None of this is a reason to avoid free domains for hobby work. It is a reason to keep the domain layer thin, documented, and easy to swap.

Migration Checklist for Near-Zero Downtime

The goal is to move from old-free-domain to new-free-domain without a visible gap for visitors. The trick is to run both names in parallel for a while and cut over only when the new one is fully verified.

1. Register the new name and set up records first

Register the new subdomain and configure its records to point at the same origin as the old one. Your host should accept both hostnames—GitHub Pages and Cloudflare Pages both let you attach multiple custom domains to one project, which is what makes this painless.

2. Verify the new name end to end

Check DNS, then check HTTPS and content:

import ssl
import urllib.request

def verify_site(url: str, expected_marker: str) -> bool:
    """Fetch a URL with certificate validation and confirm a marker string is present."""
    ctx = ssl.create_default_context()
    try:
        with urllib.request.urlopen(url, context=ctx, timeout=10) as resp:
            body = resp.read().decode("utf-8", errors="replace")
            status = resp.status
    except Exception as exc:
        print(f"{url}: FAILED ({exc})")
        return False

    ok = status == 200 and expected_marker in body
    print(f"{url}: status={status} marker_found={expected_marker in body}")
    return ok

verify_site("https://newname.dpdns.org", "your-site-title")

Do not skip this. A CNAME that resolves but serves a certificate error or a 404 from the wrong project is worse than a domain that has not propagated yet.

3. Audit the old domain for hardcoded references

Anything that embeds the old hostname needs updating: canonical tags, Open Graph URLs, sitemap entries, RSS feeds, API base URLs in frontend code, and webhook callbacks. A quick scan over a built static site:

from pathlib import Path

OLD = "oldname.dpdns.org"
NEW = "newname.dpdns.org"

def find_references(root: str) -> None:
    """Scan built output for the old hostname and report file and line."""
    for path in Path(root).rglob("*"):
        if not path.is_file() or path.suffix not in {".html", ".xml", ".json", ".js", ".css"}:
            continue
        try:
            text = path.read_text(encoding="utf-8", errors="ignore")
        except OSError:
            continue
        for lineno, line in enumerate(text.splitlines(), start=1):
            if OLD in line:
                print(f"{path}:{lineno}: {line.strip()[:120]}")

find_references("public")

Fix these in source, rebuild, and redeploy. If you rely on canonical URLs for search indexing, update them before the cutover so the new domain is the one being indexed.

4. Keep the old domain alive as a redirect

Leave the old records in place and configure your host to redirect the old hostname to the new one with a 301. This preserves inbound links and search signals. On GitHub Pages you get this by keeping both custom domains attached and adding a redirect at the edge; on Cloudflare you can do it with a redirect rule.

5. Cut over traffic sources

Update anything you control that points at the old name: bookmarks in docs, links in README files, profile URLs, and any CI jobs that curl the site for smoke tests. This is the slow part, and it is why step 4 matters—the redirect covers the tail.

6. Retire the old name later, not immediately

Keep the redirect for at least a few months. Free domains can be reclaimed if the provider changes policy, so do not assume the redirect will live forever. The point is to make the new name fully self-sufficient before the old one disappears.

Keeping the Next Migration Cheap

Two habits make all of this much easier:

Put records in a zone you control. Delegate NS from the free subdomain to a DNS provider you can export from. When the suffix changes, you repoint one delegation instead of rebuilding every record.
Never hardcode the hostname. Use a single config value for the site origin and derive canonical URLs, sitemaps, and feeds from it. The scan script above exists because this rule gets broken; a build-time constant means it only has to be changed once.

Free domains are a reasonable choice for hobby infrastructure as long as you treat them as a pointer you can move. The providers will keep changing; your build process should not care.

More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-21 08:46:52

Short

Pair	Signal	Price	Take Profit	Stop Loss	R/R
JCT	热点跟涨	$0.0047	$0.0043	$0.0048	1:1.9

1 picks selected. Scanner runs every 15 minutes.

Cross-Posting Automation: One Article, Three Platforms

mattleeee — Tue, 20 Oct 2026 01:01:44 +0000

Liquid syntax error: 'raw' tag was never closed

Generating Technical Blog Posts with the DeepSeek API

mattleeee — Mon, 19 Oct 2026 01:01:25 +0000

Liquid syntax error: Unknown tag 'endraw'

Publishing a Static Site to GitHub Pages Without git

mattleeee — Sun, 18 Oct 2026 01:01:33 +0000

Why Skip git at All?

Sometimes git isn't an option. Corporate proxies strip the smart HTTP protocol. A build agent has no credential helper and no working clone. You're inside a sandbox that only exposes HTTPS to api.github.com. The repository still needs to receive files, and GitHub Pages still needs to rebuild.

The Contents API gives you a REST path to the same effect: read a file, write a file, get a commit. It's slower than git push and it produces one commit per file, but it works from anywhere you can make an HTTPS request. This post covers the mechanics that make it behave well: SHA comparison to avoid no-op commits, a deterministic directory walk, rate-limit handling, and a verification read after every write.

The Blob SHA Is Not a Hash of the File

The first trap is assuming the sha field returned by the Contents API is sha1(file_bytes). It isn't. It's the git object ID of the blob, which is the SHA-1 of a header plus the content:

sha1(b"blob " + str(len(content)).encode() + b"\x00" + content)

That header is why you can compute the remote file's SHA locally without any network call. If your computed value matches the sha from GET /contents/{path}, the file on GitHub is byte-identical to your local copy and the PUT can be skipped entirely.

import hashlib

def git_blob_sha(content: bytes) -> str:
    header = b"blob " + str(len(content)).encode("ascii") + b"\x00"
    return hashlib.sha1(header + content).hexdigest()

This one function is the core of an idempotent deploy. Everything else is plumbing around it.

Why the skip matters

Without the comparison, every deploy rewrites every file. GitHub treats each PUT as a new commit, even when the content is unchanged, so you get:

A commit log full of "Update index.html" entries with zero diff.
A Pages build triggered per commit, so a 200-file site fires 200 builds.
Rate-limit budget consumed on writes that changed nothing.

With the comparison, a redeploy of an unchanged site makes zero PUT calls. Only files that actually changed produce commits.

Reading the Current State

GET /repos/{owner}/{repo}/contents/{path} returns either a single object (for a file) or a list (for a directory). For a file you want three fields: sha, content, and encoding. The content field is base64 with newlines embedded, so strip whitespace before decoding if you need to compare bytes rather than SHAs.

import base64
import requests

API = "https://api.github.com"

def get_file(session, owner, repo, path, ref="main"):
    url = f"{API}/repos/{owner}/{repo}/contents/{path}"
    r = session.get(url, params={"ref": ref})
    if r.status_code == 404:
        return None
    r.raise_for_status()
    data = r.json()
    if isinstance(data, list):
        raise ValueError(f"{path} is a directory, not a file")
    content = base64.b64decode(data["content"].replace("\n", ""))
    return {"sha": data["sha"], "content": content}

A 404 means the file doesn't exist at that ref yet, which is the normal case on first deploy. Treat it as None, not as an error.

Walking the Tree Deterministically

Directory iteration order differs across filesystems, and it differs again between runs on the same machine. That matters because your deploy log, your commit order, and any partial-failure recovery all depend on a stable sequence. Sort explicitly.

from pathlib import Path

def iter_files(root: Path, ignore=(".git", ".github", "__pycache__")):
    for p in sorted(root.rglob("*")):
        if not p.is_file():
            continue
        rel = p.relative_to(root)
        if any(part in ignore for part in rel.parts):
            continue
        yield rel.as_posix(), p.read_bytes()

Sorting on the full relative path string gives you a depth-first order that's stable across platforms. rglob returns paths, but you sort after filtering, so the order doesn't depend on os.scandir behavior.

Two details worth pinning down: use as_posix() for the API path so Windows separators never leak into URLs, and read bytes rather than text. Line-ending normalization or encoding guesses will change the blob SHA and defeat the comparison.

The Push

The PUT endpoint takes message, content (base64), and sha — but only when updating. Omitting sha on an existing file returns 409 Conflict, which is GitHub's way of stopping you from clobbering a concurrent write. Including a stale sha also fails, so the read and write need to be close together.

def push_file(session, owner, repo, path, content, current, message=None):
    url = f"{API}/repos/{owner}/{repo}/contents/{path}"
    local_sha = git_blob_sha(content)
    if current is not None and current["sha"] == local_sha:
        return {"status": "unchanged", "sha": local_sha}

    body = {
        "message": message or f"deploy: {path}",
        "content": base64.b64encode(content).decode("ascii"),
    }
    if current is not None:
        body["sha"] = current["sha"]

    r = session.put(url, json=body)
    r.raise_for_status()
    return {"status": "pushed", "sha": r.json()["content"]["sha"]}

The response echoes the new blob SHA under content.sha. In principle it equals local_sha. In practice, don't assume — verify it (next section).

Base64 encoding

base64.b64encode produces standard base64 with = padding, which is what the API expects. The API tolerates newlines in the payload but there's no reason to add them. For large files, the entire payload goes in a JSON body, so the practical ceiling is around 1 MB per file for the standard Contents API — above that you need the Git Data API (blobs, trees, commits) instead.

Verifying the Write

A 200 from PUT means GitHub accepted the request. It does not prove the blob on the default branch matches what you sent — webhooks, branch protection rules, or an intervening push can all change the outcome. Re-fetching the SHA is cheap and closes the loop.

def deploy_file(session, owner, repo, path, content, ref="main"):
    current = get_file(session, owner, repo, path, ref=ref)
    result = push_file(session, owner, repo, path, content, current)
    if result["status"] == "unchanged":
        return result

    check = get_file(session, owner, repo, path, ref=ref)
    if check is None or check["sha"] != git_blob_sha(content):
        raise RuntimeError(
            f"verification failed for {path}: "
            f"expected {git_blob_sha(content)}, got {check and check['sha']}"
        )
    return result

The read-after-write costs one extra request per changed file. On a deploy where most files are unchanged, that's a handful of requests. On a first deploy of a large site, it doubles the request count — worth it, because a silent mismatch means your Pages site is serving stale content and you won't know until someone notices.

Rate Limiting

Authenticated requests get 5,000 per hour against the REST API. A 300-file site with verification and a first-time full deploy costs roughly 300 GETs (returning 404 for new files), 300 PUTs, and 300 verification GETs — about 900 requests. Two deploys like that in an hour and you're at a third of your budget. Repeated deploys with the SHA skip cost 300 GETs each, since every file is still read.

Handle 403 and 429 explicitly. GitHub returns 403 with X-RateLimit-Remaining: 0 when exhausted, and Retry-After on secondary limits.

import time

def request_with_backoff(session, method, url, **kwargs):
    for attempt in range(5):
        r = session.request(method, url, **kwargs)
        if r.status_code in (403, 429):
            retry_after = r.headers.get("Retry-After")
            wait = int(retry_after) if retry_after else 2 ** attempt
            time.sleep(wait)
            continue
        return r
    return r

Wire this into get_file and push_file by replacing session.get / session.put with request_with_backoff. Cap the retries — a hard-failing deploy should stop, not spin.

If you're near the limit, the cheapest optimization is a single GET /repos/{owner}/{repo}/git/trees/{ref}?recursive=1 call to fetch every blob SHA at once, then compare locally and only GET/PUT the files that differ. That collapses the read phase from N requests to 1.

The Commit-Per-File Tradeoff

Every PUT creates a commit. There's no way around that through the Contents API alone. The consequences:

History noise. A 50-file change becomes 50 commits. If you care about readable history, batch the message (deploy: <path> [3/50]) or accept it.
Pages rebuilds. Pages builds on push to the publishing branch. Sequential commits can queue builds or cancel earlier ones. If your site takes 90 seconds to build, a 200-file deploy can take a long time to settle.
Atomicity. A failure halfway through leaves the repo in a mixed state — some files new, some old. There's no transaction. Design the deploy to be resumable: because the SHA skip is idempotent, re-running the deploy after a failure picks up exactly where it stopped.

If any of those are unacceptable, the Git Data API is the answer: create blobs, build a tree, create one commit, update the ref. That's a single commit and a single Pages build, at the cost of more code and a wider blast radius if you get the tree wrong. For small sites and infrequent deploys, per-file commits are fine.

Putting It Together

def deploy_site(root: Path, owner: str, repo: str, ref: str = "main"):
    session = requests.Session()
    session.headers.update({
        "Authorization": f"Bearer {token}",
        "Accept": "application/vnd.github+json",
        "X-GitHub-Api-Version": "2022-11-28",
    })

    stats = {"pushed": 0, "unchanged": 0}
    for path, content in iter_files(root):
        result = deploy_file(session, owner, repo, path, content, ref=ref)
        stats[result["status"]] += 1
        print(f"{result['status']:9} {path}")
    return stats

Run it twice. The second run should report every file as unchanged and make zero writes. If it doesn't, your comparison is broken — most often because you're reading text instead of bytes, or because something in the build pipeline is rewriting files with different line endings between runs.

That idempotency is the whole point. It makes the deploy safe to retry, keeps the commit log honest, and keeps Pages from rebuilding for nothing.

More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-18 08:46:32

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
AGT	弱做多	$0.0843	$0.0906	$0.0822	1:3.0
BTW	短平快做多	$2.3242	$2.3939	$2.2661	1:1.2

2 picks selected. Scanner runs every 15 minutes.

Publishing to Juejin Through Its Web API: The mark_content Lesson

mattleeee — Sat, 17 Oct 2026 01:01:34 +0000

Automating content publishing is a common task for developer-advocates and technical writers. Juejin (掘金) is a popular Chinese developer community, and like many modern web apps, it relies on an internal HTTP API for its editor. That API is undocumented and changes without notice. This post is a post-mortem of a silent regression: drafts saved with empty bodies and a publish call returning err_no=1010. The fix turned out to be a single field name, but finding it required a disciplined reverse-engineering loop that is worth writing down.

The Symptom

The publisher had been running for months. It did three things:

POST /api/v1/draft/save with the article body.
POST /api/v1/draft/publish to push the draft live.
Log the returned article_id and URL.

Then, without any code change on our side, every run produced a draft with an empty body. The publish step returned:

{"err_no": 1010, "err_msg": "草稿内容为空"}

Translated: "draft content is empty." The draft existed, the title was correct, but the body was blank. The API had added a requirement: the Markdown source must be sent in a field called mark_content, separate from the rendered HTML field the client had been sending.

Reverse-Engineering the Regression

The first move with any undocumented API is to stop guessing and capture what the real client does. Open Juejin in a browser, open DevTools, go to the Network tab, and filter for Fetch/XHR. Then perform the action manually — write a draft with some text and hit save.

You will see a request to something like:

POST https://api.juejin.cn/content_api/v1/article_draft/save

Right-click the request, choose Copy as cURL (bash), and paste it into a text editor. This gives you the full request: method, URL, headers, cookies, and the JSON body. Do the same for the publish action. Two captures are enough to reconstruct the whole flow.

The cURL for the save call looks roughly like this (truncated):

curl 'https://api.juejin.cn/content_api/v1/article_draft/save' \
  -H 'content-type: application/json' \
  -H 'cookie: sessionid=...; uuid=...' \
  -H 'user-agent: Mozilla/5.0 ...' \
  --data-raw '{"title":"test","mark_content":"# hello","brief_content":"","category_id":"1","tag_ids":[]}'

The key observation: the body contains both mark_content and, in the full capture, a html_content or similar rendered field. The old client only sent the rendered field. The new API validates mark_content and rejects the request if it is missing or empty.

Reading the Error Code

err_no=1010 is not documented. But the mapping is easy to infer once you reproduce it deliberately:

Save a draft with mark_content present and non-empty → publish succeeds.
Save a draft with mark_content omitted → publish returns 1010.
Save a draft with mark_content: "" → publish returns 1010.

That three-point experiment tells you the field is required and that empty string is treated as absent. It also tells you the failure is server-side validation, not an auth problem — a 401 or err_no in the 1000s with a different message would point elsewhere.

Replaying the Calls from Python

With the cURL captures in hand, the translation to requests is mechanical. The only care needed is header fidelity: content-type, user-agent, and the cookie header.

import json
import requests

BASE = "https://api.juejin.cn"
SAVE_URL = f"{BASE}/content_api/v1/article_draft/save"
PUBLISH_URL = f"{BASE}/content_api/v1/article_draft/publish"

HEADERS = {
    "content-type": "application/json",
    "user-agent": (
        "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
        "AppleWebKit/537.36 (KHTML, like Gecko) "
        "Chrome/122.0.0.0 Safari/537.36"
    ),
    "origin": "https://juejin.cn",
    "referer": "https://juejin.cn/",
}


def save_draft(session: requests.Session, title: str, markdown: str,
               category_id: str, tag_ids: list[str]) -> str:
    payload = {
        "title": title,
        "mark_content": markdown,      # the field that broke everything
        "brief_content": markdown[:100],
        "category_id": category_id,
        "tag_ids": tag_ids,
    }
    resp = session.post(SAVE_URL, headers=HEADERS,
                        data=json.dumps(payload), timeout=30)
    resp.raise_for_status()
    body = resp.json()
    if body.get("err_no") != 0:
        raise RuntimeError(f"save failed: {body}")
    return body["data"]["id"]


def publish_draft(session: requests.Session, draft_id: str) -> str:
    payload = {"draft_id": draft_id, "column_ids": []}
    resp = session.post(PUBLISH_URL, headers=HEADERS,
                        data=json.dumps(payload), timeout=30)
    resp.raise_for_status()
    body = resp.json()
    if body.get("err_no") != 0:
        raise RuntimeError(f"publish failed: {body}")
    return body["data"]["article_id"]

Two details matter here. First, data=json.dumps(payload) is used instead of json=payload. Both work in most cases, but the explicit form makes the serialization visible and avoids surprises if you later need to control separators or encoding. Second, the session object is reused so cookies persist across the save and publish calls — publishing a draft that was just saved in a different session will fail.

The Cookie Problem in Windows Batch Launchers

The publisher was launched from a .bat file, which is where the next bug lived. The cookie header contains sessionid=...; uuid=.... In a batch file, % is the escape character for environment variables. If any cookie value contains a %, CMD will try to expand it and mangle the string. Worse, the mangling is silent — you get a 401 or a mysterious err_no with no hint about why.

The fix is to stop passing cookies as a command-line string entirely. Store them in a file and read them at runtime.

cookies.json:

{
  "sessionid": "your-session-id-here",
  "uuid": "your-uuid-here"
}

Loader:

import json
from pathlib import Path

def load_session(cookie_file: str = "cookies.json") -> requests.Session:
    data = json.loads(Path(cookie_file).read_text(encoding="utf-8"))
    session = requests.Session()
    for name, value in data.items():
        session.cookies.set(name, value, domain=".juejin.cn")
    return session

Now the .bat launcher is trivial and cannot mangle anything:

@echo off
python publish.py --draft drafts\my-post.md

If you must keep cookies in the batch file for some reason, double every % as %%. But the file approach is strictly better: it is readable, diffable, and easy to rotate when the session expires. Never commit cookies.json — add it to .gitignore. The sessionid is a full account credential.

Batch Backfill: Republishing 20 Historical Posts

Once the single-post path worked, the obvious next step was to repair the archive. Twenty posts had been published with empty bodies over the broken window. Rather than fix them by hand, a backfill script iterated over local Markdown files, matched each to its existing draft or article ID, and re-saved with mark_content populated.

The matching step is the interesting part. If you did not store the article ID at publish time, you cannot reliably match by title alone. The pragmatic approach is to store a small manifest alongside the drafts:

{
  "my-post.md": {"draft_id": "7123...", "article_id": "7124..."}
}

The backfill script then reads the manifest, loads each Markdown file, and re-saves:

import json
from pathlib import Path

def backfill(session, drafts_dir: str, manifest_path: str,
             category_id: str, tag_ids: list[str]) -> None:
    manifest = json.loads(Path(manifest_path).read_text(encoding="utf-8"))
    ok, failed = 0, []

    for filename, entry in manifest.items():
        path = Path(drafts_dir) / filename
        if not path.exists():
            failed.append((filename, "missing file"))
            continue

        markdown = path.read_text(encoding="utf-8")
        title = markdown.splitlines()[0].lstrip("# ").strip()

        try:
            new_id = save_draft(session, title, markdown,
                                category_id, tag_ids)
            entry["draft_id"] = new_id
            ok += 1
        except Exception as exc:  # noqa: BLE001
            failed.append((filename, str(exc)))

    Path(manifest_path).write_text(
        json.dumps(manifest, ensure_ascii=False, indent=2),
        encoding="utf-8",
    )
    print(f"backfilled {ok}, failed {len(failed)}")
    for name, reason in failed:
        print(f"  {name}: {reason}")

Two operational notes from running this against 20 posts:

Rate-limit yourself. Insert a time.sleep(2) between iterations. Twenty rapid-fire saves can trip a soft throttle, and the failure mode is an ambiguous error rather than a clean 429.
Write the manifest after every success, not at the end. If the script dies at post 14, you do not want to re-save the first 13. The version above writes once at the end for brevity; in production, flush after each item.

The run republished all 20 posts in a single pass. The only failures were two files whose titles had been edited in the CMS after the original publish, so the manifest IDs pointed at stale drafts. Those were resolved by re-creating the drafts and updating the manifest.

Lessons for Any Undocumented API

The regression was a one-field change, but the process that found it generalizes:

Capture before you theorize. Copy-as-cURL from DevTools is the ground truth. The browser is the only client the server is guaranteed to support.
Diff the capture against your payload. The missing field was visible the moment the two were placed side by side.
Reproduce the error deliberately. Vary one field at a time until you can trigger and clear the failure on demand. That is what turns err_no=1010 from a mystery into a known precondition.
Keep credentials out of shell strings. Files beat command lines, especially on Windows.
Make batch operations idempotent and resumable. Persist progress as you go.

The publisher has run clean since. The mark_content field is now set in one place, and the backfill script lives in the repo as a reusable repair tool for the next time the API quietly grows a new requirement.

More notes like this ship every week on this site.

Making .bat Files Bulletproof Across Windows Machines

mattleeee — Fri, 16 Oct 2026 01:01:28 +0000

Batch files are the glue of Windows automation and the most fragile part of it. They look trivial: a text file with a few commands, saved as .bat, double-clicked. Then you copy that file to another machine and it explodes — sometimes literally, with a console full of 'xxx' is not recognized as an internal or external command. The failures are rarely about the commands. They are about bytes: the encoding, the line endings, the codepage, and the invisible characters that a text editor added without asking.

This post covers the hardening rules I now apply to every .bat I ship, why each rule exists, and how each one has broken a real deployment. At the end there is a byte-level verification script you can run before shipping.

The failure mode is always the same

A batch file is not parsed by a compiler. cmd.exe reads it line by line, byte by byte, and interprets what it finds. There is no schema, no lint, no type system. If a byte is unexpected, cmd either ignores it, misinterprets it, or feeds it to a command that does something you did not intend.

That means every rule below is really the same rule: control the bytes.

Rule 1: Pure ASCII content

The single most common cause of "works on my machine" batch failures is non-ASCII text. A Chinese comment, a curly quote pasted from a document, an em dash — all of it becomes a liability the moment the file leaves the editor that created it.

Here is the mechanism. cmd.exe reads the batch file using the console's current codepage, not a fixed encoding declared in the file. On a Chinese Windows install that is typically 936 (GBK). On a US install it is 437 or 1252. On a modern developer machine it might be 65001 (UTF-8) if someone ran chcp 65001 earlier in the session. The same bytes are decoded differently depending on the machine.

A UTF-8 Chinese comment like :: 检查目录 becomes three bytes per character. Under codepage 936, those bytes decode into different characters, and — worse — some byte sequences contain 0x5C (\) or 0x7C (|), which cmd treats as syntax. I have seen a comment line turn into a pipe and split a command in half. The error message pointed at a line that looked completely innocent.

The fix is boring: keep the .bat pure ASCII. Comments in English, no accented characters, no smart quotes, no emoji. If you genuinely need localized documentation, put it in a companion .txt file next to the script:

deploy.bat
deploy.notes.zh-CN.txt

The .txt is never executed, so its encoding does not matter to cmd. The .bat stays safe. This also has a nice side effect: the script becomes readable to anyone on the team regardless of their Windows language settings.

If you must print localized text from the batch file itself, see Rule 3 — but keep the file content ASCII even then, and emit the localized strings via a separate mechanism.

Checking for non-ASCII bytes

This is the check I run most often, because it catches the problem before it ever reaches a machine:

from pathlib import Path

def find_non_ascii(path: Path) -> list[tuple[int, int, int]]:
    """Return (line_no, col_no, byte) for every non-ASCII byte."""
    hits = []
    data = path.read_bytes()
    line, col = 1, 0
    for b in data:
        if b == 0x0A:
            line += 1
            col = 0
            continue
        col += 1
        if b > 0x7F:
            hits.append((line, col, b))
    return hits

for line, col, byte in find_non_ascii(Path("deploy.bat")):
    print(f"line {line}, col {col}: 0x{byte:02X}")

Even one hit is enough to reject the file. There is no upside to a non-ASCII byte in a .bat.

Rule 2: CRLF line endings, always

Unix line endings (LF, 0x0A) are the second classic killer. cmd.exe expects CRLF (0x0D 0x0A). What happens with bare LF depends on the command, which is what makes it so nasty.

Simple commands often work. But goto labels, if blocks with parentheses, and for loops can behave unpredictably, because cmd reads ahead and the missing carriage return shifts its notion of where a line ends. The symptom is usually a goto that jumps to nowhere, or a block that executes the wrong branch. I once spent an afternoon on a for /f loop that silently skipped its body on one machine and worked on another — the only difference was that the broken file had been checked out with core.autocrlf=false.

The root cause is usually Git. If .bat files are committed with LF and checked out on Linux or macOS, they arrive on Windows with LF. The fix is a .gitattributes entry:

*.bat text eol=crlf
*.cmd text eol=crlf

That single line prevents an entire class of incidents. Add it before you have batch files in the repo, not after.

Verifying line endings in Python

from pathlib import Path

def check_line_endings(path: Path) -> list[int]:
    """Return line numbers that end with bare LF (no preceding CR)."""
    bad = []
    data = path.read_bytes()
    line = 1
    i = 0
    while i < len(data):
        b = data[i]
        if b == 0x0A:
            if i == 0 or data[i - 1] != 0x0D:
                bad.append(line)
            line += 1
        i += 1
    return bad

Note that this flags LF at the very start of the file too, which is correct — a batch file should not begin with a bare newline.

Rule 3: Do not rely on the console codepage

chcp changes the active console codepage, and it is tempting to write chcp 65001 at the top of a script so that UTF-8 output works. This is a trap for two reasons.

First, chcp 65001 historically broke more than it fixed on older Windows versions, and behavior still varies by build. Second, and more importantly, the codepage at the time cmd reads the file is what matters for parsing — changing it inside the script only affects output after that point. If the file itself contains non-ASCII bytes, you have already lost before chcp runs.

The working pattern is:

Keep the .bat pure ASCII (Rule 1).
If you must print localized text, call chcp as the very first command, and only when needed.
Better: do not print localized text from the batch file at all. Let a Python helper do it.

@echo off
setlocal
rem Pure ASCII only. No chcp unless localized output is required.
python "%~dp0report.py" %*
if errorlevel 1 (
    echo report.py failed with code %errorlevel%
    exit /b %errorlevel%
)
endlocal

The Python helper handles its own encoding, which is far more controllable:

import sys

def safe_print(text: str) -> None:
    """Print to stdout, degrading gracefully if the console can't encode it."""
    try:
        print(text)
    except UnicodeEncodeError:
        enc = sys.stdout.encoding or "ascii"
        print(text.encode(enc, errors="replace").decode(enc))

This keeps the batch file dumb and portable, and puts the encoding decisions where they can actually be tested.

Rule 4: Quote every path, and derive paths from `%USERPROFILE%`

Unquoted paths break on spaces. This is old news, but the subtler problem is which paths you use. Hardcoding C:\Users\Alice works until the script runs as a service account, on a machine where the profile is on D:, or under a different user. Derive everything from environment variables:

@echo off
setlocal
set "APPDIR=%USERPROFILE%\AppData\Local\MyTool"
set "LOGDIR=%APPDIR%\logs"
if not exist "%APPDIR%" mkdir "%APPDIR%"
if not exist "%LOGDIR%" mkdir "%LOGDIR%"

Every expansion is quoted. %USERPROFILE% can contain spaces (rare but legal), and %APPDIR% certainly can. An unquoted mkdir %APPDIR% with a space in the path creates two directories in the current folder and then fails on the second one.

Rule 5: Use `%~dp0` for self-locating paths

Scripts that reference sibling files — a .py helper, a config, a log — should locate them relative to the script, not relative to the current working directory. %~dp0 expands to the drive and path of the batch file itself, including the trailing backslash.

@echo off
setlocal
python "%~dp0report.py" --config "%~dp0config.json"

Without this, double-clicking the .bat from Explorer works (cwd is the script folder), but running it from a shortcut or a scheduler task with a different working directory fails. I have seen a scheduled task run at 3 AM with cwd C:\Windows\System32 and fail to find its own config. %~dp0 fixes it permanently.

One caveat: %~dp0 already ends with a backslash, so "%~dp0\file" produces a double backslash. It usually works, but write "%~dp0file" instead.

Rule 6: No trailing spaces after `set "VAR=value"`

This is the one that costs people hours because the output looks correct. Consider:

set "MODE=prod" 
echo Mode is [%MODE%]

There is a single trailing space after prod on the first line. cmd includes it in the value. The variable is now prod, not prod. Every comparison against prod fails, every path built from it has a trailing space, and the echo shows [prod ] — which is easy to miss when you skim the console.

The quoted form set "VAR=value" protects against trailing spaces before the assignment and against special characters in the value, but it does not protect against a space inside the quotes. There is no way to see it in a normal editor. This is exactly the kind of thing a byte-level check should catch.

from pathlib import Path
import re

SET_RE = re.compile(rb'^\s*set\s+"([^=]+)=([^"]*)"\s*$', re.IGNORECASE)

def find_trailing_space_in_set(path: Path) -> list[tuple[int, bytes, bytes]]:
    """Flag `set "VAR=value "` where value ends with a space."""
    hits = []
    for i, line in enumerate(path.read_bytes().splitlines(), start=1):
        m = SET_RE.match(line)
        if m and m.group(2).endswith(b" "):
            hits.append((i, m.group(1), m.group(2)))
    return hits

Any hit is a bug. Fix the source line; do not try to compensate downstream.

The pre-ship verification script

All of the above collapses into one script. Run it in CI or as a pre-commit hook. It reads the file as bytes, which is the only honest way to inspect a batch file.

from pathlib import Path
import re
import sys

SET_RE = re.compile(rb'^\s*set\s+"([^=]+)=([^"]*)"\s*$', re.IGNORECASE)

def audit_bat(path: Path) -> list[str]:
    problems = []
    data = path.read_bytes()

    # 1. Non-ASCII bytes
    line, col = 1, 0
    for b in data:
        if b == 0x0A:
            line += 1
            col = 0
            continue
        col += 1
        if b > 0x7F:
            problems.append(f"line {line}: non-ASCII byte 0x{b:02X}")

    # 2. Bare LF line endings
    line = 1
    for i, b in enumerate(data):
        if b == 0x0A:
            if i == 0 or data[i - 1] != 0x0D:
                problems.append(f"line {line}: bare LF (expected CRLF)")
            line += 1

    # 3. Trailing space inside set "VAR=value "
    for i, raw in enumerate(data.splitlines(), start=1):
        m = SET_RE.match(raw)
        if m and m.group(2).endswith(b" "):
            problems.append(f"line {i}: trailing space in set \"{m.group(1).decode()}=...\"")

    # 4. Unquoted %USERPROFILE% expansions (heuristic)
    for i, raw in enumerate(data.splitlines(), start=1):
        stripped = raw.strip()
        if stripped.startswith(b"rem") or stripped.startswith(b"::"):
            continue
        for var in (b"%USERPROFILE%", b"%APPDIR%"):
            if var in raw and b'"' not in raw:
                problems.append(
                    f"line {i}: {var.decode()} used without quotes"
                )

    return problems

if __name__ == "__main__":
    failed = False
    for p in sys.argv[1:]:
        path = Path(p)
        issues = audit_bat(path)
        if issues:
            failed = True
            print(f"{path}:")
            for issue in issues:
                print(f"  {issue}")
    sys.exit(1 if failed else 0)

The heuristic in check 4 is deliberately crude — it flags lines that mention those variables without any quotes at all. It is not a parser, but it catches the mistakes that actually happen. Extend the variable list as your scripts grow.

Run it like this:

python audit_bat.py deploy.bat install.bat

Exit code 1 means do not ship. Wire it into CI and the class of failures described here mostly disappears.

What this buys you

None of these rules are clever. They are the residue of real incidents: a comment that split a command, a goto that missed its label, a scheduled task that could not find its config, a variable with an invisible trailing space. Each one is cheap to prevent at the byte level and expensive to debug at runtime.

The habit that matters is reading batch files as bytes, not as text. Once you do that, the fixes are obvious. More notes like this ship every week on this site.

Daily Picks

The following pairs are selected from the multi-timeframe trend scanner (Gate.io futures) and are for technical-analysis study only — not investment advice.
Data updated: 2026-10-16 08:46:41

Long

Pair	Signal	Price	Take Profit	Stop Loss	R/R
ZEREBRO	高质量标的做多	$0.051	$0.0541	$0.0494	1:2.0
ARC	短平快做多	$0.0359	$0.037	$0.035	1:1.2

2 picks selected. Scanner runs every 15 minutes.

PowerShell on Legacy Chinese Windows: UTF-8 with BOM or Bust

mattleeee — Wed, 14 Oct 2026 01:01:34 +0000

Windows PowerShell 5.1 does not guess encodings. When it loads a .ps1 file, it looks for a byte-order mark. If it finds one, it decodes accordingly. If it does not, it falls back to the system's ANSI code page — and on a Chinese Windows install, that code page is GBK (CP936). Every non-ASCII character in your script is then decoded through the wrong table, and you get mojibake that does not throw an error. It just quietly changes what your script means.

This is the story of how a path check turned into a week of debugging, and the one rule that fixed it: every .ps1 file ships as UTF-8 with BOM and CRLF endings.

The symptom

A deployment script had a line like this:

$configDir = Join-Path $env:ProgramData "我的工具\配置"
if (-not (Test-Path $configDir)) {
    New-Item -ItemType Directory -Path $configDir -Force | Out-Null
}

On the developer's machine it worked. On the build server it created a directory named something like æˆ‘çš„å·¥å…· — or sometimes it created nothing at all, and Test-Path returned $false for a path that clearly existed when inspected manually in Explorer.

The reason is straightforward once you see the bytes. The file was saved by an editor as UTF-8 without BOM. PowerShell 5.1 read it as GBK. The UTF-8 bytes for 我 are E6 88 91. Decoded as GBK, those three bytes are a valid two-byte character plus a stray byte, producing a different string entirely. PowerShell then passed that corrupted string to the Windows API, which dutifully created a directory with a nonsense name. Test-Path on the intended name failed because the intended name was never used.

No exception. No warning. Just a different program than the one you wrote.

Detecting it: hex-dump the first bytes

The fastest way to confirm the diagnosis is to look at the raw bytes at the start of the file.

# Show the first 8 bytes of a script file
$bytes = [System.IO.File]::ReadAllBytes(".\deploy.ps1")[0..7]
($bytes | ForEach-Object { $_.ToString("X2") }) -join ' '

Interpretation:

EF BB BF — UTF-8 with BOM. PowerShell 5.1 decodes correctly.
FF FE — UTF-16 LE with BOM. Also fine.
FE FF — UTF-16 BE with BOM. Fine, rare.
Anything else — no BOM. PowerShell 5.1 falls back to the ANSI code page. On a GBK system that means GBK.

If you see E6 88 91 at the start of a file containing 我, you have a BOM-less UTF-8 file, and 5.1 is about to misread it.

A quick Python check across a whole directory is often easier than doing this by hand:

from pathlib import Path

BOMS = {
    b"\xef\xbb\xbf": "utf-8-sig",
    b"\xff\xfe": "utf-16-le",
    b"\xfe\xff": "utf-16-be",
}

def sniff(path: Path) -> str:
    head = path.read_bytes()[:4]
    for bom, enc in BOMS.items():
        if head.startswith(bom):
            return enc
    # No BOM. Try strict UTF-8; if it fails, it is almost certainly ANSI/GBK.
    try:
        path.read_bytes().decode("utf-8")
        return "utf-8 (NO BOM) <-- 5.1 will misread this"
    except UnicodeDecodeError:
        return "ansi/gbk (NO BOM)"

for ps1 in Path("scripts").rglob("*.ps1"):
    print(f"{ps1}: {sniff(ps1)}")

Run this on any legacy repo and the offending files surface immediately. On a Chinese Windows box, the mix of "utf-8 (NO BOM)" and "ansi/gbk" files is the whole problem in one listing.

Why BOM-less UTF-8 is a trap on Chinese Windows

The default encoding for reading files in PowerShell 5.1 is not UTF-8. It is [System.Text.Encoding]::Default, which maps to the active ANSI code page. On an English system that is CP1252. On a Simplified Chinese system it is CP936 (GBK). On a Japanese system, CP932. The behavior of the same script file therefore depends on the locale of the machine running it.

This is the part that makes the bug so expensive. The script is not wrong on every machine — it is wrong on machines whose code page differs from the one the author implicitly assumed. A BOM removes the ambiguity: it tells the parser "this is UTF-8" regardless of locale.

The same trap applies to Get-Content and Set-Content in 5.1. Get-Content without -Encoding uses the ANSI default; Set-Content without -Encoding writes ANSI. Round-tripping a UTF-8 file through those cmdlets silently converts it to GBK.

Bulk-converting a repo to UTF-8 with BOM

You can fix a directory of scripts with PowerShell itself. The trick is to read the bytes, decode them with the correct source encoding, and write them back with a BOM.

# convert-to-utf8bom.ps1
param(
    [Parameter(Mandatory)][string]$Root,
    [string]$SourceEncoding = "utf-8"   # use "gb2312" for legacy GBK files
)

$utf8Bom = New-Object System.Text.UTF8Encoding($true)

Get-ChildItem -Path $Root -Recurse -Filter *.ps1 | ForEach-Object {
    $path  = $_.FullName
    $bytes = [System.IO.File]::ReadAllBytes($path)

    # Already has a UTF-8 BOM? Skip.
    if ($bytes.Length -ge 3 -and $bytes[0] -eq 0xEF -and $bytes[1] -eq 0xBB -and $bytes[2] -eq 0xBF) {
        Write-Host "skip (already BOM): $path"
        return
    }

    $text = [System.Text.Encoding]::GetEncoding($SourceEncoding).GetString($bytes)

    # Normalize line endings to CRLF.
    $text = $text -replace "`r`n", "`n" -replace "`n", "`r`n"

    [System.IO.File]::WriteAllText($path, $text, $utf8Bom)
    Write-Host "converted: $path"
}

Two things matter here. First, WriteAllText with a UTF8Encoding($true) instance emits the BOM; UTF8Encoding($false) does not. Second, you must know the source encoding. If a file is genuinely GBK, decoding it as UTF-8 will throw or produce replacement characters. The Python sniffer above tells you which is which before you run the conversion.

The same job in Python, which is often more convenient in CI:

import sys
from pathlib import Path

def to_utf8_bom(path: Path, source_encoding: str) -> None:
    raw = path.read_bytes()
    if raw.startswith(b"\xef\xbb\xbf"):
        return
    text = raw.decode(source_encoding)
    text = text.replace("\r\n", "\n").replace("\n", "\r\n")
    path.write_bytes(b"\xef\xbb\xbf" + text.encode("utf-8"))

for p in Path(sys.argv[1]).rglob("*.ps1"):
    try:
        to_utf8_bom(p, "utf-8")
    except UnicodeDecodeError:
        to_utf8_bom(p, "gbk")
    print(f"converted {p}")

The fallback to gbk on UnicodeDecodeError handles mixed repos where some files were saved as ANSI by an older editor. Once every file is UTF-8 with BOM, the ambiguity is gone.

PowerShell 7 behaves differently

PowerShell 7 (Core) changed the defaults. Get-Content and the script loader default to UTF-8 without requiring a BOM, and Set-Content writes UTF-8 without BOM by default. So the same BOM-less file that 5.1 mangles will load correctly in 7.

That is a feature, and also a trap. If your scripts run under both 5.1 and 7 — which is common on servers that still have Windows PowerShell installed alongside Core — you need the file to work under the stricter reader. UTF-8 with BOM works in both. BOM-less UTF-8 works only in 7. So the rule is not "use UTF-8"; the rule is "use UTF-8 with BOM" if any 5.1 host might touch the file.

Note also that PowerShell 7 still honors the BOM when present, so adding it does not break anything under Core.

A pre-flight check before you ship

The fix is cheap; the discipline is what fails. Put a validator in your build or pre-commit step so a BOM-less file never reaches a 5.1 host.

#!/usr/bin/env python3
"""Fail the build if any .ps1 is not UTF-8 with BOM and CRLF."""
import sys
from pathlib import Path

def check(path: Path) -> list[str]:
    errors = []
    raw = path.read_bytes()

    if not raw.startswith(b"\xef\xbb\xbf"):
        errors.append("missing UTF-8 BOM")
    else:
        try:
            raw[3:].decode("utf-8")
        except UnicodeDecodeError as exc:
            errors.append(f"not valid UTF-8: {exc}")

    # CRLF check: reject lone LF outside of the final byte.
    body = raw[3:] if raw.startswith(b"\xef\xbb\xbf") else raw
    if b"\n" in body.replace(b"\r\n", b""):
        errors.append("contains LF-only line endings; expected CRLF")

    return errors

failed = False
for p in sorted(Path(".").rglob("*.ps1")):
    errs = check(p)
    if errs:
        failed = True
        for e in errs:
            print(f"{p}: {e}")

sys.exit(1 if failed else 0)

Drop this into CI as a gate. The failure message is specific enough that the fix is obvious: save the file as UTF-8 with BOM, convert line endings to CRLF, commit.

If you want the validator to auto-fix instead of fail, reuse the conversion functions above and re-run the check. In a pre-commit hook, auto-fixing is usually the better default — it removes the human from the loop entirely.

The rule

Every .ps1 file that might ever run under Windows PowerShell 5.1 must be saved as UTF-8 with BOM and CRLF line endings. BOM-less UTF-8 is a valid encoding and a broken script on a GBK system. UTF-16 also works but is unusual in a repo and confuses diffs and tooling. Stick to UTF-8 with BOM.

The failure mode — silent mojibake, locale-dependent behavior, no exception — is what makes this worth a hard rule rather than a convention. A linter catches it in a second; a human catches it after a week of Test-Path returning $false on a path that exists.

More notes like this ship every week on this site.