Pin URL Query Encoding Before One Client Extract

Pin URL query encoding before you extract any HTTP client helper. Callers often depend on plus signs, sorted keys, and hidden safe characters. A cleanup extract can change every signed request without failing unit tests.

Messy repos bury urlencode calls across adapters, retries, and debug dumps. Each site passes a slightly different doseq, safe, or quote_via. Characterization tests freeze those contracts before any move.

The contract you actually ship

Query strings are not a cosmetic serialization detail. Signature schemes hash the exact bytes on the wire. Caching layers key on the encoded form, not the dict.

Pin at least five observable fields before you touch production clients.

1. Space encoding chooses plus signs or percent-twenty sequences.
2. Key order follows caller order or an implicit sort.
3. Sequence folding depends on the doseq boolean flag.
4. Extra unescaped characters come from the safe argument.
5. Empty values are omitted, set to key-equals, or bare keys.

Skip this list and the extract will look green. Downstream auth will still break on the first replay.

Decision table for one extract

Use this table before you allow a helper to exist. Every row is a frozen contract, not a preference.

Observable	Keep as-is	Forbidden during extract
Space	Current quote_via	Switching plus and percent-twenty
Order	Current pair order	Sorting keys for readability
Sequences	Current doseq	Joining lists with commas
Safe chars	Current safe set	Unescaping slashes or tildes
Empty	Current omission rules	Dropping empty keys

The extract may move code between modules only. It may not change any table cell.

Step 1: inventory every query builder

Search the repo with a boring, exact pattern. Do not start in a generated client tree.

rg -n "urlencode|quote_plus|urlparse|parse_qsl" --type py

Record file, line, and call keywords in a short table. Treat request params kwargs as another query builder. Treat f-string query assembly as a third builder.

Stop if you find more than one encoding policy. Two policies means two helpers later, not one.

Count hits per package before you pick a target. Start with the module that builds signed URLs. Leave generated SDK wrappers for a later pin.

Step 2: record goldens as strings you can diff

The next block is a proposed characterization harness. It writes JSON goldens you can commit.

# tools/pin_query_goldens.py
from __future__ import annotations

import json
from pathlib import Path
from urllib.parse import quote, urlencode

GOLDEN = Path("tests/goldens/query_encoding.json")

SAMPLES = [
    {"id": "spaces", "params": [("q", "hello world"), ("n", "1")]},
    {"id": "unicode", "params": [("q", "café")]},
    {"id": "reserved", "params": [("path", "/a/b"), ("q", "a+b")]},
    {"id": "seq", "params": [("tag", ["a", "b"]), ("q", "x")]},
    {"id": "empty", "params": [("q", ""), ("keep", "1")]},
    {"id": "safe_tilde", "params": [("q", "~user/name")]},
]


def encode_variants(params):
    as_dict = {}
    for key, value in params:
        if key in as_dict:
            prev = as_dict[key]
            as_dict[key] = prev + [value] if isinstance(prev, list) else [prev, value]
        else:
            as_dict[key] = value
    return {
        "urlencode_pairs": urlencode(params),
        "urlencode_doseq": urlencode(params, doseq=True),
        "urlencode_quote_via_quote": urlencode(params, quote_via=quote),
        "urlencode_doseq_quote": urlencode(params, doseq=True, quote_via=quote),
        "urlencode_safe_tilde": urlencode(params, safe="~"),
        "dict_urlencode": urlencode(as_dict),
        "dict_urlencode_doseq": urlencode(as_dict, doseq=True),
    }


def main() -> None:
    rows = []
    for sample in SAMPLES:
        encoded = encode_variants(sample["params"])
        rows.append({"id": sample["id"], "params": sample["params"], **encoded})
    GOLDEN.parent.mkdir(parents=True, exist_ok=True)
    GOLDEN.write_text(json.dumps(rows, indent=2, ensure_ascii=False) + "\n")
    print(f"wrote {GOLDEN} rows={len(rows)}")


if __name__ == "__main__":
    main()

Run it once on the current interpreter. Commit the JSON file as the encoding pin.

python tools/pin_query_goldens.py
git add tests/goldens/query_encoding.json
git commit -m "test: pin urllib.parse query encoding goldens"

Label this run as a local characterization, not a published benchmark. Rerun it only when the interpreter series changes.

Passing a dict with list values stringifies the brackets. Pair lists with doseq true preserve repeated keys. Your goldens must include both input shapes.

Step 3: fail closed on any encoding drift

Add a test that rereads the golden file. Do not regenerate golden files inside the CI job.

# tests/test_query_encoding_goldens.py
from __future__ import annotations

import json
from pathlib import Path

from tools.pin_query_goldens import encode_variants

GOLDEN = Path("tests/goldens/query_encoding.json")


def test_query_encoding_matches_pinned_goldens():
    rows = json.loads(GOLDEN.read_text())
    assert rows, "golden file must not be empty"
    for row in rows:
        got = encode_variants([tuple(item) for item in row["params"]])
        for key, expected in got.items():
            assert row[key] == expected, f"{row['id']}.{key}"

Run the characterization test before any production edit. A red test means your pin is wrong. Fix the golden pin before any refactor.

Store the golden path in one constant. Duplicate golden paths cause silent test splits. Keep the writer and reader on the same helper.

Step 4: pin one production call site

Goldens of urllib.parse alone are not enough. Next, wrap the messiest production builder with tests.

# tests/test_legacy_search_url.py
from urllib.parse import parse_qsl, urlsplit

from payments.client import build_search_url  # existing messy function


def test_legacy_search_url_query_bytes():
    url = build_search_url(
        host="https://example.test",
        q="hello world",
        tags=["a", "b"],
        path="/v1/items",
    )
    parts = urlsplit(url)
    assert parts.scheme == "https"
    assert parts.netloc == "example.test"
    assert parts.path == "/v1/items"
    # Pin the encoded query, not a parsed dict.
    assert parts.query == "q=hello+world&tags=a&tags=b"
    assert parse_qsl(parts.query, keep_blank_values=True) == [
        ("q", "hello world"),
        ("tags", "a"),
        ("tags", "b"),
    ]

Adjust the expected string to whatever the repo already emits. Do not correct that encoded string here.

Parse the query only after the raw string assertion. Parsed dicts hide plus and order bugs. Raw query strings are the real product contract.

Step 5: one extract, same query string

Only now extract one helper with identical defaults. Copy the urlencode kwargs into a named function.

# payments/query.py
from urllib.parse import urlencode


def build_query(params):
    """Proposed extract. Keep urlencode kwargs identical."""
    return urlencode(params, doseq=True)  # doseq=True is the pinned contract

Then change exactly one production call site. Leave every other query builder completely untouched.

# inside payments/client.py — single call site
from payments.query import build_query


def build_search_url(host, q, tags, path):
    query = build_query([("q", q), ("tags", tags)])
    return f"{host}{path}?{query}"

Re-run both test files after the extract. The encoded query string must remain unchanged after extract.

pytest tests/test_query_encoding_goldens.py tests/test_legacy_search_url.py -q

If the assertion on parts.query fails, revert the extract. Do not patch the golden to match new taste.

Put the chosen kwargs in a one-line comment. Future editors should see doseq without code archaeology. Those comments are cheaper than a broken signature.

Step 6: add a byte checksum for signed clients

String equality is enough for most internal apps. Signed clients should also pin UTF-8 digest bytes. The next test is a proposed extra lock.

# tests/test_search_query_digest.py
import hashlib
from pathlib import Path
from urllib.parse import urlsplit

from payments.client import build_search_url

PIN = Path("tests/goldens/search_query.sha256")


def test_search_query_sha256():
    url = build_search_url(
        host="https://example.test",
        q="hello world",
        tags=["a", "b"],
        path="/v1/items",
    )
    query = urlsplit(url).query.encode("utf-8")
    digest = hashlib.sha256(query).hexdigest()
    if not PIN.exists():
        PIN.write_text(digest + "\n")
    assert digest == PIN.read_text().strip()

Keep this checksum next to the query golden. Do not hash parsed dictionaries in this checksum. Hash the exact query component text encoded as UTF-8.

Write the pin file on one local run only. Commit it before reviewers inspect the extract. Later runs must not rewrite the digest file.

Failure analysis: three innocent diffs

The first innocent diff sorts keys for stable logs. Sorting changes cache keys and HMAC bases. Revert that sort during the extract window.

The second innocent diff switches quote_plus to quote. Spaces then stop being encoded as plus signs. Many form endpoints will reject the new client.

The third innocent diff joins list values with commas. The doseq flag instead repeats the same key. Analytics backends treat those shapes as different filters.

Where a free model may help

After the goldens fail closed, a model can draft the extract. It should not choose safe, doseq, or sort order.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. Use that workspace to propose the single-function move only.

Paste the failing test and the current function, not the whole monolith. Reject any diff that edits urlencode keyword arguments. Reject any diff that silently sorts query keys.

Reject any diff that touches a second call site. This is a proposed workflow, not a measured product study. No model names are required for the workflow.

Limitations

These pins capture CPython urllib.parse for one interpreter. They do not prove wire compatibility with Go, Java, or browsers. They do not cover IDNA hosts, IPv6 brackets, or fragment handling.

urlencode dict input drops repeated keys unless you pass pairs. Goldens must use the same input type as production.

Compare query strings as text in the golden file. Encode UTF-8 later if signatures hash raw bytes.

Do not treat plus-versus-percent as a style war. RFC 3986 prefers percent-twenty for the space character.

Form encoding prefers plus signs for spaces. Your repository already chose one encoding convention.

Relative URLs, matrix params, and semicolon separators stay out of scope. So do httpx params mutations and requests PreparedRequest paths. Pin those stacks with their own goldens later.

Who should not use this approach

Do not use this extract path for public API query contracts. Ship an explicit versioned encoder for those contracts instead. Do not use it when tests cannot run without network.

Do not use it to standardize several services in one commit. Skip it if the client already has a frozen HTTP cassette. Extend that cassette before extracting any helper.

Skip it for binary query values or non-UTF-8 payloads. Skip it when legal holds require bit-identical traffic replays without tests. In that case, capture packets before any extract.

Close the loop

Keep the golden JSON in the same commit as the extract. Reviewers then see exact bytes, not stated intent.

A later cleanup may still want sorted keys. That work is a second change with a new pin.

The smallest safe change is the one that leaves parts.query identical. If that string changes, the extract was not safe.