DEV Community: Zhang Enquan

I Tokenized 10,000 Real Prompts: The "4 Characters 1 Token" Rule Is Off by More Than 20% on 4 Out of 10 of Them

Zhang Enquan — Wed, 14 Oct 2026 13:21:06 +0000

For three months I priced every LLM feature with one number: 4 characters ≈ 1 token.

It is everywhere. It is in blog posts, in pricing calculators, in the back of my head when somebody asks "roughly how much will this cost". It feels safe because it is conservative: it rounds up the token count, so your estimate lands above the real bill.

Last week I stopped trusting it and measured it. 10,000 real samples, real tokenizer, no synthetic text.

The rule is bad per prompt. It is fine for the monthly total. Knowing which of those two you are doing turns out to be the whole game.

How I measured it

I did not want synthetic text, because synthetic text is where benchmarks go to lie. The corpus is 521 real posts pulled from the dev.to public API, across ai, webdev, python, programming, javascript and opensource.

Each post was stripped of front matter, then split into:

prose — paragraphs with code fences removed
code — the contents of fenced blocks
structured — blocks that are JSON-ish or log-shaped

That gave a pool I sampled 10,000 from: 7,055 prose, 2,669 code, 275 structured. Every sample was encoded with cl100k_base through tiktoken, one at a time, and I compared the real count against len(text) / 4.

Two honest caveats before the numbers:

This is English technical writing. I got exactly one CJK sample in the whole corpus, so I make no claim about Chinese, Japanese, Korean or emoji-heavy text. If your traffic is multilingual, treat every number here as a lower bound on how wrong the rule is.
The mix is heavy on prose because that is what dev.to posts are. Do not read the "all" column as "the internet".

What the rule does to a single prompt

Measure	Result
Median error	+11.7% (the rule over-counts)
Median absolute error	16.3%
Samples off by more than 20%	40.5%
of those, over-counted	33.3%
of those, under-counted	7.2%
5th percentile	−23.9%
95th percentile	+46.8%

So on a single prompt, the rule is a coin flip that is usually wrong by more than a fifth. If you have ever sized a context window with chars / 4 and wondered why the request still got truncated, that 7.2% under-count is why. The rule is mostly conservative, but "mostly" is not "always", and the failure mode is the expensive one: you packed too much and the call failed.

The part I did not expect: the monthly bill was basically right

Here is the aggregate over all 10,000 samples:

2,715,542 characters
646,092 real tokens
→ 4.203 characters per token

Run chars / 4 over the whole pile and you get 678,885 tokens against a true 646,092. That is +5.08%.

Read those two sections together, because they look like they contradict each other. They do not:

Per prompt: 40% of samples are off by more than 20%.
In aggregate: the batch is off by 5%.

The errors cancel. Short samples and long samples, prose and code, they all pull the average toward 4.2 from different directions. So the rule is not "wrong" in the way people say it is wrong. It is unusable per call and acceptable per month. If you are budgeting a feature, chars / 4 will get you a number that is about 5% high, which is fine. If you are deciding whether one specific prompt fits in a 128k window, it will get you a number that is wrong one time in three.

Short prompts are where it actually breaks

I bucketed by length and this is the finding I now care about most:

Sample length	n	Median chars/token	Median abs error	Off by >20%
0–200 chars	5,455	4.500	17.5%	43.6%
200–500 chars	3,448	4.537	16.6%	41.4%
500–1500 chars	951	4.186	10.6%	24.6%
1500–4000 chars	146	4.018	4.8%	4.8%

The shorter the text, the worse the rule. Below 200 characters — which is most chat prompts, most system messages, most tool descriptions — it is off by more than 20% on 43.6% of samples. Above 1,500 characters it is nearly fine.

That is the opposite of what intuition suggests. You would think a tiny string is easy to estimate. It is not: a single rare word, an unusual identifier or one emoji can swallow a dozen characters into three tokens, and there is no averaging to save you.

Content type moves the constant

Type	n	Median chars/token	Weighted chars/token	Median error with /4
Prose	7,055	4.577	4.375	+14.4%
Code	2,669	4.156	4.010	+3.9%
Structured (JSON, logs)	275	3.778	3.877	−5.6%

Structured is the one that should worry you. With chars / 4 it under-counts: median error −5.6%, and 21.1% of structured samples were under-counted by more than 20%.

Which means: if your app emits JSON and you set a hard token cap by dividing characters by four, you are setting that cap too low roughly a fifth of the time. You will reject responses you could have afford. That is the mirror image of the truncation bug, and it is the one nobody writes about.

So which number should you use?

I tried a range of constants against the same 10,000 samples:

Constant	Median abs error	Median bias
4.0	16.3%	+11.7%
4.25	13.5%	+5.2%
4.47	11.9%	−0.03%
4.5	11.9%	−0.7%
4.75	12.1%	−5.9%

4.47 is the unbiased one — it splits the difference almost perfectly. 4.5 is indistinguishable in practice and easier to remember. But notice the ceiling: even the best constant still has a median absolute error of 11.9%. No single number fixes this. The distribution is too wide for a constant to win.

My rules now:

Budget with 4.5, bill with a real tokenizer. 4.5 for the spreadsheet, actual encode() before anything that costs money or hits a limit.
Never set a hard cap with chars / 4 on structured output. That is the one direction where the rule fails cheap instead of safe.
Anything under 200 characters, do not estimate. Measure it once, cache the number. These are the samples where the rule is worst and they are also the ones you reuse a million times, so one measurement pays for itself immediately.

Counting it for real

If you want to see where a tokenizer actually splits your text — which is the only way to build intuition for why résumé and resume are not the same price — I built a small counter that shows the split boundaries per word: LLM Token Counter. It runs in the browser, no upload.

If you would rather run it yourself in Python or Node, the walkthrough is here: How to count GPT tokens.

And for the case where characters are the unit you actually care about: Word & character counter.

One question

I can tell you about English technical prose. I cannot tell you about yours.

If you have measured this on your own traffic — especially non-English, chat logs, or agent traces — what constant did you land on? I would like to know whether 4.2 holds outside dev.to, or whether this whole thing is even more corpus-dependent than I think.

The raw data (10,000 rows: kind, chars, tokens, ratio, error) is in the repo with the script that produced it. Rerun it on your own text; that takes five minutes and beats trusting my constant.

How to Count GPT Tokens Before You Hit the API

Zhang Enquan — Tue, 06 Oct 2026 06:48:50 +0000

You cannot control what you do not measure — and GPT APIs bill you for something you never see.

Every request is priced, rate-limited, and truncated by token count, not by words, characters, or number of calls. If you only learn the real number from the API response, you already paid for it. Counting GPT tokens before you send gives you three things: a cost you can quote, a context window you know you fit inside, and a prompt you can trim while trimming is still cheap.

Here is how to do it in Python and JavaScript, which encoding to use, and what your request actually contains.

Tokens are not words

A token is a subword unit from a byte-pair encoding (BPE) tokenizer. Common English words usually become one token. Less common ones get split — tokenization becomes token + ization, two tokens for one word.

For rough planning, English prose runs about 4 characters per token, or roughly 0.75 words per token. That average breaks the moment the input stops looking like English prose:

Identifiers, URLs, file paths — a UUID is several tokens; every path segment splits separately
Code — indentation, punctuation, and operators each cost tokens
Numbers — digits are often split into groups, so 2026 may not be one token
CJK text — Chinese characters commonly cost 1–2 tokens each, so character heuristics undercount badly
Emoji and accents — expand into multiple tokens once UTF-8 encoded
Pretty-printed JSON — indentation across a big object adds up fast; collapse it before the call

You also cannot estimate in the other direction. A character or word count gives you a ceiling, not the billed number.

Which encoding your model uses

Counts differ between tokenizer generations. Using the wrong one gives you a plausible number that is off by 10–20% on English text and much more on anything else.

Encoding	Vocab (approx.)	Used by
`o200k_base`	~200,000	gpt-4o, gpt-4o-mini, gpt-4.1 family, gpt-5 family, o1, o3, o3-mini, o4-mini
`cl100k_base`	~100,000	gpt-4, gpt-4-turbo, gpt-3.5-turbo, text-embedding-3-small/large
`p50k_base`	~50,000	text-davinci-002/003, code-davinci-002
`r50k_base` (gpt2)	~50,000	gpt-2, davinci, curie, babbage, ada

You rarely need to memorize it — tiktoken resolves the encoding from the model name by prefix matching, so anything starting with gpt-5-, gpt-4o-, gpt-4.1-, o1-, o3-, or o4-mini- resolves to o200k_base.

import tiktoken

for model in ("gpt-4o", "gpt-4o-mini", "gpt-4.1", "gpt-5-mini", "o3-mini", "gpt-4", "gpt-3.5-turbo"):
    print(model, tiktoken.encoding_for_model(model).name)

Count GPT tokens in Python

pip install tiktoken

import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o-mini")   # resolves to o200k_base
text = open("prompt.txt", encoding="utf-8").read()
print(len(enc.encode(text)), "tokens")

It runs offline — no API key, no network request after the encoding file is cached.

When a count surprises you, look at how the string actually splits:

ids = enc.encode("Unbelievable")
print(ids)
print([enc.decode_single_token_bytes(t) for t in ids])
# [59026, 17536]  ->  [b'Un', b'believable']

Counting a whole chat request

Each message carries structural overhead beyond its content. This approximation follows OpenAI's cookbook: roughly 3 tokens per message, plus 1 if you set a name, plus 3 priming the assistant reply.

import tiktoken

def count_chat_tokens(messages, model="gpt-4o-mini"):
    try:
        enc = tiktoken.encoding_for_model(model)
    except KeyError:
        enc = tiktoken.get_encoding("o200k_base")   # fallback for new model names

    tokens_per_message, tokens_per_name, total = 3, 1, 0
    for message in messages:
        total += tokens_per_message
        for key, value in message.items():
            if isinstance(value, str):
                total += len(enc.encode(value))
            if key == "name":
                total += tokens_per_name
    total += 3  # assistant reply primer
    return total

Treat it as a planning figure. The authoritative count is always the usage object in the response — log it and reconcile against your estimate.

Count GPT tokens in JavaScript

Counting where the prompt is built lets you warn users before they send something enormous, without shipping their text anywhere.

npm install js-tiktoken

import { encodingForModel } from "js-tiktoken";

const enc = await encodingForModel("gpt-4o");
console.log(enc.encode("Count me before I cost you money.").length, "tokens");

gpt-tokenizer is the other common option and ships per-model entry points, which helps when bundle size matters:

import { encode } from "gpt-tokenizer";
const n = encode(text).length;

The tokenizer is just a data table and a merge algorithm, so none of this needs a network call.

What your request actually contains

The user message is usually the smallest part of the bill. All of this counts against the same context window and the same input price:

System prompt — resent every request, so length here is a permanent tax
Conversation history — grows quadratically with turns if replayed in full
Tool/function schemas — twenty tools with detailed descriptions can occupy thousands of tokens even when none are called
Retrieved context — every pasted chunk
Reasoning tokens — on reasoning models these occupy context and bill as output
The reply — output usually costs several times more per token than input, and max_tokens is a cap, not a budget

Budgeting the context window

The window is shared, not per-message:

free_tokens = context_window
            - system + history + tools + retrieved_chunks
            - max_tokens_you_allow_for_the_reply

If that lands near zero you get either a rejected request or silently dropped messages — your "memory" vanishes mid-conversation. Keep 10–15% slack so unusual inputs do not tip you over. Better: size retrieval from the count. After counting the fixed parts, whatever remains is exactly how many tokens of context you can afford.

Turning tokens into money

cost = (input_tokens / 1_000_000) * input_price
     + (output_tokens / 1_000_000) * output_price

Standard-tier OpenAI list prices, October 2026 (always re-check the official pricing page):

Model	Input / 1M	Cached input / 1M	Output / 1M
gpt-5-nano	$0.05	$0.005	$0.40
gpt-5-mini	$0.25	$0.025	$2.00
gpt-4o-mini	$0.15	$0.075	$0.60
gpt-4o	$2.50	$1.25	$10.00
o3	$2.00	$0.50	$8.00

Concrete case: 10,000 support answers per day, 600-token prompt, 300-token reply.

gpt-4o-mini: 6M input × $0.15 = $0.90/day + 3M output × $0.60 = $1.80/day → $2.70/day (~$81/month)
gpt-4o: 6M × $2.50 = $15.00/day + 3M × $10.00 = $30.00/day → $45.00/day (~$1,350/month)

Same prompt, same answer quality for a routing task, 16× difference. This is why counting comes before model selection — you cannot compare prices without knowing volume.

Six traps that inflate prompts

Replaying the whole history — summarize old turns or use a rolling window
Minifying JSON by hand — whitespace is real money at scale; strip it, drop unused field names
Registering every tool on every request — send only this turn's schemas
Trusting one "average tokens per request" number — track mean, p90, p99; the p99 breaks your budget
Trusting character heuristics for non-English text — 4 chars/token is an English figure
Setting max_tokens to the max — an uncapped reply is the classic surprise bill

Checklist before every production request

Count GPT tokens with the encoding of the model you actually serve
Include system prompt, history, tools, and retrieved chunks — not just the user message
Leave headroom: prompt + expected reply < context window
Log response usage and compare against your estimate weekly
Hard-cap both input length and max_tokens
Re-check prices quarterly and after any model-family switch

This post is the written-up version of how I actually budget prompts before shipping anything that calls an LLM. The full guide with extra sections is on my site: How to Count GPT Tokens Before You Hit the API.

If you work in the browser, two tools from the same toolkit are useful alongside this: a JSON Formatter & Validator for collapsing whitespace out of payloads, and a Word & Character Counter for the fast ceiling estimate. Both run locally — nothing uploaded.

How do you budget tokens in your pipeline — pre-flight counting, response usage logging, or something else?

Cloudflare Just Shipped Post-Quantum at Internet Scale. Do You Need to Change Your Tools?

Zhang Enquan — Mon, 14 Sep 2026 13:19:23 +0000

On September 8, 2026, Cloudflare flipped the switch on post-quantum TLS origin handshakes across its edge. Two days later, on September 10, 1.1.1.1 enabled post-quantum DNSSEC powered by NIST's ML-DSA-44 algorithm. The numbers are not small: the TLS deployment alone now covers roughly 45 billion daily connections.

If you saw that headline and immediately opened your code editor to grep for md5 and sha256 — this post is for you. If you saw it and shrugged because you only use hashing for "checksumming" — this post is also for you.

If that's you — this is roughly what your terminal looks like right now:

$ rg -n "md5|sha256" src/ --color=always | head -20
src/main.c:1:    fprintf(stderr, "ERROR: Unsupported hash function: %s\n", hash_type);
src/CMakeLists.txt:21:    target_compile_options(${PROJECT_NAME} PRIVATE
                                                     -Werror=deprecated-declarations)
src/Rust/Cargo.toml:45:    version = "0.1.1"
src/Rust/Cargo.toml:46:    md5    = "0.10.0"
src/Rust/Cargo.toml:47:    sha256 = "0.9.0"
src/Rust/src/lib.rs:120:   let hash = md5::compute(data);
src/Rust/src/lib.rs:11:    let hash = sha2::Sha256::digest(data);
src/scripts/build.sh:15:    if ! command -v openssl   &> /dev/null; then
src/scripts/build.sh:15:    if ! command -v sha256sum &> /dev/null; then
src/scripts/build.sh:12:        echo "ERROR: Neither openssl nor sha256sum found!" >&2
src/scripts/util.sh:35:    md5sum    "$file" | awk '{print $1}'
src/scripts/util.sh:36:    sha256sum "$file" | awk '{print $1}'

(That's not a snippet from a real repo, but the panic it captures is.)

The short version: most of the tools you already use are fine. The panic comes from confusing hashes with encryption, and from using encryption when you actually needed a checksum.

Let's untangle that.

The panic that doesn't help

Here's what happened in a few Discord servers I read this week:

"Should I migrate from MD5 to SHA-256? Grover's algorithm makes SHA-256 only 128-bit secure anyway. Should I just go to SHA-3?"

"Should I rotate all my UUIDs to UUIDv7?"

"Is base64 still safe?"

These are reasonable questions if you don't know the answer. They are also three different categories of mistake.

MD5, SHA-1, and SHA-256 are hash functions. They produce a fixed-size fingerprint from any input. The reason they exist is not to keep secrets — it's to verify that data didn't change in transit. A 1GB ISO file you downloaded can be checksummed with any of them; the question is "did the bytes arrive intact", not "can someone read the bytes".

Base64 is an encoding, not encryption. It exists so that binary data can travel through channels (JSON, URL paths, email) that only support text. There is no key. There is no "secure" mode. Anyone who tells you base64 is "encrypted" is selling you snake oil.

UUID v4 is a random identifier, 122 bits of randomness. It's not a hash, not an encryption — it's a label. Whether quantum computers threaten it depends on whether you used a cryptographic random number generator (good) or Math.random() (you have bigger problems than Grover).

So when Cloudflare ships post-quantum TLS, here's what's actually happening at the protocol level:

The TLS handshake negotiates session keys using a key exchange algorithm that a quantum computer cannot efficiently break.
Everything inside that TLS tunnel — your HTTP traffic, your JSON, your base64-encoded blobs — is still protected by classical symmetric ciphers (AES-256-GCM, ChaCha20) and authenticated by classical hashes (SHA-256, SHA-384).
The risk model is "harvest now, decrypt later": an adversary records encrypted traffic today, waits for a cryptographically-relevant quantum computer (CRQC), then decrypts it. That's the threat post-quantum key exchange is closing.

Your MD5 checksum tool does not enter this picture.

What each tool is actually for

Since I built AI Subtools — a no-fluff browser-side toolbox — here's the honest guide. Same tool, different intent:

Tool	What it does	Threat model	Replace with PQC?
CRC32 Checksum Generator	Fast integrity check on small files / packets	Accidental corruption	No — error-detection, not security
MD5 Hash Generator	Legacy checksum, fingerprint for deduplication	Accidental corruption	No, for the same reason
SHA256 Hash Calculator	Strong fingerprint for file integrity / commit hashes	Accidental corruption	Optional — SHA-256 is fine for non-adversarial checks
Base64 Encoder/Decoder	Binary ↔ text transport	None — it's encoding	No — never was security
URL Encoder/Decoder	Make strings safe for URLs	None	No
UUID v4 Generator	Random 122-bit ID	Collision, predictability	No, if you use `crypto.getRandomValues`
Password Strength Check	Estimate entropy of a passphrase	Dictionary attack	The length matters, not the hash
Random Password Generator	High-entropy secrets	Predictability	No, if CSPRNG-backed
Encrypt/Decrypt (AES-256)	Symmetric encryption, browser-side	Adversary steals ciphertext	No — AES-256 is post-quantum-safe

The last row is the only one where a quantum computer would matter at all, and even then: Grover's algorithm gives a quadratic speedup, which means AES-256 effectively becomes AES-128 — still considered computationally infeasible to break. NIST standardized it for exactly that reason.

So when do you actually need post-quantum?

There are exactly three situations where PQC migration matters for the average developer:

You run a TLS server. Migrate your server library to support hybrid Kyber/X25519 key exchange (OpenSSL 3.5+, BoringSSL, rustls 0.23+). Cloudflare already did this for you if your users connect via their edge.
You sign software or firmware updates. Classic ECDSA signatures are quantum-broken. Migrate to ML-DSA-44 / Dilithium for new releases; consider re-signing historical binaries if your threat model includes harvest-now-decrypt-later.
You embed long-lived secrets in firmware or code signing certificates. Anything with a 10+ year shelf life needs a PQC plan now.

Everything else — your file checksums, your base64-encoded JSON payloads, your UUIDs, your SHA-256 commit hashes — is already fine.

A simple decision tree

When you reach for one of these tools, ask one question: am I protecting against bit-flips, or against an adversary?

Bit-flips only (corrupted download, broken pipe, accidental edit) → CRC32, MD5, SHA-256. Any of them. CRC32 is fastest; SHA-256 is most defensible.
Adversary who can read but not modify → AES-256 encryption. Done.
Adversary who can read and modify → authenticated encryption (AES-GCM) or HMAC over the ciphertext.
Adversary with a quantum computer who recorded your traffic years ago → PQC key exchange at the protocol layer (already handled by Cloudflare, AWS, GCP, Fastly, Akamai in 2026).

That's it. No framework. No vendor lock-in.

FAQ

Q: Is MD5 broken?
Yes, but not for checksumming. Collision attacks against MD5 mean you can craft two different files with the same MD5 — useful for forging signatures, useless for catching accidental corruption. Use MD5 for cache keys, deduplication, and legacy file verification. Don't use it for digital signatures — that hasn't been safe since 2008.

Q: Should I switch to SHA-3 for file integrity?
No reason to. SHA-256 is fine for non-adversarial integrity. The actual quantum risk lands on digital signatures and key exchange, not on file checksums.

Q: Is base64 encryption?
No. It is an encoding. There is no key. Anyone — including this page's CSS — can decode it. If someone tells you they "encrypted it with base64", they did not.

Q: Is UUID v4 quantum-safe?
UUID v4 is 122 bits from a CSPRNG. Brute-forcing 122 bits is computationally infeasible even with Grover's algorithm — you'd need ~2^61 operations, which is more atoms than in a kilogram of lead. As long as your UUID generator uses a cryptographic random source, you're fine.

Q: What about AES-256?
Post-quantum-safe by design. Grover halves effective key length to 128 bits. NIST confirmed AES-256 is acceptable for top-secret data through the post-quantum era.

Closing thought

Cloudflare's PQC deployment this week was an infrastructure milestone, not a consumer panic trigger. The right response is to make sure your TLS library supports hybrid post-quantum key exchange — not to throw out your hash functions.

If you're picking a tool for a one-off task today, pick the one that matches the threat model, not the trend.

— Built and maintained at aisubtools.xyz — 40+ free browser-side tools for developers and creators.

Open-sourcing my dev tools as plain HTML files (no npm, no build)

Zhang Enquan — Thu, 10 Sep 2026 07:48:49 +0000

```Last month I shipped three developer utilities — a Base64 encoder, a Unix
timestamp converter, and a CSS minifier. Each one runs100% in your browser.
No servers, no uploads, no analytics, no tracking.
\

Yesterday I open-sourced them on GitHub. This post is the story of why I
built them this way and why I open-sourced them at all.

The problem with "free online tools"

Most "free online tools" secretly upload your input to a server, process it
there, and hope nobody keeps a copy. That is a terrible default.

Try this experiment: paste 你好世界 🌍 into the most popular online Base64
encoder. About half of them will return å¥½ä¸ç•Œ — broken output that no
longer decodes back to what you typed. Why? Because most sites treat text as
Latin-1 by accident.

Even when they work correctly, you're trusting that the site:

won't log your input
won't sell it to a third party
won't get breached next year
won't disappear and take your workflow with it

Open source fixes all four.

What I shipped

I run aisubtools.xyz — a collection of free online
utilities where nothing ever leaves your browser. Yesterday I extracted
three of them into a standalone open-source repo:

🔗 github.com/enquannacc/aisubtools-browser-tools

Tool	What it does
Base64 encoder/decoder	Handles UTF-8 correctly via `TextEncoder` / `TextDecoder`. No mojibake.
Unix timestamp converter	Bidirectional conversion with local and UTC time, plus a live clock.
CSS minifier	Regex-based whitespace stripper with before/after size savings.

MIT licensed. No npm install. No build step. No dependencies. Open the file,
use it, close the tab. Done.

The architecture: one HTML file per tool

Every tool in the repo is a single self-contained HTML file. Here's the
pattern I used across all three:


html
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Base64 Encode / Decode — Browser Tool</title>
  <style>
    /* ~100 lines of CSS — nothing fancy, system fonts, CSS variables */
 </style>
</head>
<body>
  <h1>Base64 Encode / Decode</h1>
  <p>Runs 100% in your browser — nothing is uploaded anywhere.</p>
  <textarea id="input" placeholder="Type or paste here"></textarea>
  <button id="convert">Convert</button>
  <textarea id="output" readonly></textarea>
  <script>
    const input = document.getElementById('input');
    const output = document.getElementById('output');
    const convert = document.getElementById('convert');

    let mode = 'encode';

    function utf8ToBase64(str) {
      return btoa(String.fromCharCode(
        ...new TextEncoder().encode(str)
      ));
    }

    function base64ToUtf8(b64) {
      const bytes = Uint8Array.from(atob(b64), c => c.charCodeAt(0));
      return new TextDecoder().decode(bytes);
    }

    convert.addEventListener('click', () => {
      try {
        output.value = mode === 'encode'
          ? utf8ToBase64(input.value)
          : base64ToUtf8(input.value.trim());
      } catch (e) {
        output.value = 'Error: ' + e.message;
      }
    });
  </script>
</body>
</html>

Why %20 Keeps Appearing in Your URLs: A Practical Guide to URL Encoding

Zhang Enquan — Sun, 06 Sep 2026 14:19:43 +0000

You paste a URL into your browser, hit Enter, and suddenly it's full of %20, %3F, and %26. Or worse — your API call fails because a query parameter contains a space or an & you didn't encode.

URL encoding trips up everyone eventually. The good news: it's much simpler than it looks. This is a practical guide to reading percent-encoded URLs, knowing when encoding happens automatically, and fixing the bugs it causes.

Why URL encoding exists at all

A URL has to be plain ASCII — but the real world isn't. Spaces, emoji, Chinese characters, ampersands... all of them need to travel through a URL without breaking its structure.

The solution: any character that isn't URL-safe gets replaced with a % followed by its hexadecimal byte value.

A space becomes %20 (hex 20 = 32 = space in ASCII)
& becomes %26
? becomes %3F

The ? and & are the interesting ones — because those characters do have a special job in URLs. That's the key to understanding the whole system.

The two categories you must not confuse

Every character in a URL falls into one of two buckets:

1. Reserved characters — they are part of the URL's structure:

: / ? # [ ] @ ! $ & ' ( ) * + , ; =

These have meaning: ? starts the query string, & separates parameters, # starts the fragment. If your data contains one of these characters, it must be encoded — otherwise the URL's structure gets corrupted.

2. Unsafe / non-ASCII characters — spaces, control characters, non-Latin text, emoji. They must also be encoded because URLs are transmitted as ASCII.

Everything else (letters, digits, -, _, ., ~) passes through untouched.

That's the entire mental model. When you see %26 in a URL, it doesn't mean "the website hates you" — it means the original data contained a literal & that had to be protected so it wouldn't be mistaken for a parameter separator.

The cheat sheet you'll actually use

Encoded	Character	Where you'll meet it
`%20`	space	Search queries, file names
`%2F`	`/`	Paths inside query params
`%3F`	`?`	A literal `?` in data
`%26`	`&`	Values containing `&`
`%3D`	`=`	Values containing `=`
`%25`	`%`	The classic double-encode bug
`%2B`	`+`	Legacy form encoding of spaces

About %25: this is my favorite interview question. If you see %2520 in a URL, the original data was %20 — someone encoded an already encoded string. The % became %25, leaving 20 intact. Double-encoding bugs almost always trace back to a URL being encoded twice by two different layers (your code + a library).

When encoding happens automatically (and when it doesn't)

This is where bugs sneak in. The browser encodes some things for you, but not everything, and not everywhere.

The browser address bar is forgiving. Type https://example.com/search?q=hello world and the browser will quietly encode the space for you. This hides bugs during manual testing that blow up when the same URL is requested programmatically.

fetch() and <a href> are NOT forgiving in the same way. Spaces and non-ASCII characters in a query string can produce malformed requests or, worse, silently work in one browser and fail in another.

Form submissions (GET forms) encode spaces as +. That's why hello+world and hello%20world can both mean "hello world", depending on which layer produced them. This is also why decoding a form-encoded string with the wrong decoder gives you literal + signs scattered through your data.

The JavaScript functions, once and for all

In JavaScript you have four tools, and picking the wrong one is a rite of passage:

// encodeURIComponent — for a VALUE going into a query string
encodeURIComponent("fish & chips? yes")
// "fish%20%26%20chips%3F%20yes"
// ✅ Encodes & and ? — the structure is protected

// encodeURI — for a WHOLE URL that's already assembled
encodeURI("https://example.com/search?q=hello world")
// "https://example.com/search?q=hello%20world"
// ✅ Leaves :// and ? alone, encodes the space

// decodeURIComponent / decodeURI — the reversals
decodeURIComponent("fish%20%26%20chips")
// "fish & chips"

The rule of thumb:

Encoding one value (about to insert it into a query string)? → encodeURIComponent
Encoding a complete URL that's already well-formed except for stray spaces/unicode? → encodeURI

The mistake I see most often: using encodeURI on a value. It leaves & and ? untouched, so fish & chips slips into the URL as-is and silently truncates your query string at the &.

// ❌ Broken: value contains & and ?
"https://example.com/q?" + encodeURI("fish & chips?")

// ✅ Correct
"https://example.com/q?" + encodeURIComponent("fish & chips?")

Three real bugs you'll eventually meet

1. The truncated query string. A search for Tom & Jerry becomes ?q=Tom and everything after & becomes a new (empty) parameter. The page loads, the search just returns wrong results. Maddening until you look at the Network tab.

2. The double-encoded redirect. You pass a ?next= parameter containing a full URL. Some middleware encodes it again. The redirect target now contains %253A instead of %3A. The fix is to decode exactly once at each layer, and to be disciplined about which layer owns encoding.

3. The emoji that kills your sitemap. Non-ASCII characters in URLs are legal (via encoding), but if your CMS or export script writes raw UTF-8 into an XML sitemap without encoding, Google may start complaining. Always encode before writing URLs into structured files.

When you just need to check one string quickly

For debugging, I keep a browser-side URL encoder/decoder bookmarked — paste the string, see it encoded and decoded side by side, copy whichever form you need. It runs client-side, so tokens and internal URLs never leave the browser. Pair it with console.log and the Network tab and you can diagnose 95% of encoding bugs in a minute.

The 30-second summary

%XX is just a character's hex byte value, prefixed with %
Reserved characters (& ? = / #) must be encoded when they're data, not structure
The address bar hides encoding bugs; test with fetch or curl
encodeURIComponent for values, encodeURI for whole URLs
%25 appearing where %20 should be means something double-encoded

URL encoding stops being scary the moment you stop reading %3F as noise and start reading it as "that's a literal question mark in the data". Once that clicks, every encoded URL you see becomes just... a string with a hat on.

Questions or war stories about encoding bugs?

I Built 35+ Browser-Side Tools Because I Was Tired of Uploading My Data to Random Servers

Zhang Enquan — Wed, 02 Sep 2026 14:50:11 +0000

Every time I needed to format a JSON payload or compress an image, I found myself on some random tool site with three problems:

My data went to their server. Paste a password or an API token into a "hash generator" and you're just trusting that site not to log it.
Ads everywhere. The tool was buried under banners, and half the buttons were ad-click traps.
Sign-up walls. Want to download your converted file? Create an account first.

So I built AI SubTools — a collection of 35+ free tools where everything runs client-side in JavaScript. No uploads. No accounts. No ads.

Why client-side matters

It's not just a privacy stance. When the tool runs in your browser:

It's instant. No round-trip to a server, no queue. Image compression happens as fast as your CPU allows.
It works offline once the page is loaded.
Your data never leaves your device. For tools that touch passwords, tokens, or checksums, this should be the default, not a feature.

Some of the more interesting tools

Cron expression generator & tester — with plain-English explanations of what your expression actually does
PNG background remover — runs entirely in-browser, no upload
Robots.txt + XML sitemap generator — for the SEO-minded
EAN/UPC barcode creator — for e-commerce sellers
JSON formatter & validator, Base64 encoder/decoder, MD5/SHA-256 generators — the daily-driver classics

The technical trade-offs

The hardest parts were making image compression and background removal perform well without WASM. Canvas-based processing is fast enough for most images, but there's a real art to avoiding UI freezes on large files — chunked processing and careful memory management matter more than raw algorithm speed.

What's next

The site is a side project that grew out of my own daily frustration. New tools are added regularly, and I'm currently prioritizing:

More file-format converters
Better mobile experience for the image tools

What tools would you want added? I'd genuinely love to hear what's missing from your daily workflow — the best suggestions will get built.

P.S. Everything is free, forever. No premium tier hiding the useful stuff.