DEV Community

BeGoodTool.com profile image Joe Lin
Joe Lin for BeGoodTool.com

Posted on

Why pasted text keeps breaking search and formatting (and the regexes I ended up using to clean it)

#vue #javascript #webdev

I kept running into a boring problem that was harder to debug than it should have been: text that looked normal, but behaved wrong the moment I pasted it into a CMS, a spreadsheet, or a code comment. Search would fail. Line breaks would get weird. A heading copied from ChatGPT would drag Markdown markers along with it. Sometimes the only visible clue was that the punctuation felt slightly "off."

What finally made this manageable wasn't some big NLP trick. It was going back to the dumb, reliable layer: exact character matching. The tool I built for this is basically a pile of small, deterministic cleanups for the specific junk that copied text tends to accumulate — full-width punctuation mixed into ASCII, invisible Unicode code points, curly quotes, em dashes, leftover Markdown, and whitespace noise.

The most useful part is the invisible-character scan, not the cleaning

The piece I trust most in the whole component is the part that explicitly names which invisible characters it cares about, then counts them by code point. It's not doing a vague "this text seems suspicious" pass. It has a hard-coded inventory:

const invisibleDefs = [
  { key: "zwsp", codes: [0x200b] },
  { key: "zwnj", codes: [0x200c] },
  { key: "zwj", codes: [0x200d] },
  { key: "bomZwnbsp", codes: [0xfeff] },
  { key: "wordJoiner", codes: [0x2060] },
  { key: "softHyphen", codes: [0x00ad] },
  { key: "bidiMarks", codes: [0x200e, 0x200f, 0x202a, 0x202b, 0x202c, 0x202d, 0x202e] },
];

const codesToRegex = (codes) =>
  new RegExp(`[${codes.map((c) => "\\u" + c.toString(16).padStart(4, "0")).join("")}]`, "g");

const analyzeInvisible = (str) => {
  const breakdown = invisibleDefs.map((def) => ({
    key: def.key,
    count: (str.match(codesToRegex(def.codes)) || []).length,
  }));
  const total = breakdown.reduce((sum, row) => sum + row.count, 0);
  return { breakdown, total };
};
Enter fullscreen mode Exit fullscreen mode

I like this because it's brutally literal. U+200B isn't treated the same as U+FEFF, and bidi control marks are grouped together on purpose instead of disappearing into one generic "hidden char" bucket. That matters when you're trying to answer why text is broken, not just make it stop being broken.

It also explains why this kind of cleaner is more useful than asking an AI model to "clean up my text." LLMs are decent at rewriting sentences, but they are famously shaky at exact character-level accounting. This component is just matching code points and counting hits. If there are three soft hyphens and one BOM, it tells you there are three soft hyphens and one BOM.

A nice touch in the Vue code is that the UI doesn't just show one scary total. It filters the breakdown so only categories with nonzero counts are shown. That makes the result read like a diagnosis instead of a dump.

Full-width and half-width normalization is just a code-point offset — with a very deliberate boundary

The width-normalization logic is much simpler than most people expect:

const FULLWIDTH_CHARS_REGEX = /[！-～]/g;
const HALFWIDTH_CHARS_REGEX = /[\x21-\x7E]/g;

const toHalfWidth = (str) =>
  str.replace(FULLWIDTH_CHARS_REGEX, (ch) => String.fromCharCode(ch.charCodeAt(0) - 0xfee0));
const toFullWidth = (str) =>
  str.replace(HALFWIDTH_CHARS_REGEX, (ch) => String.fromCharCode(ch.charCodeAt(0) + 0xfee0));
Enter fullscreen mode Exit fullscreen mode

That 0xFEE0 offset is the whole trick. For the visible ASCII range, the full-width forms in Unicode are laid out at a fixed distance from their half-width counterparts, so the conversion can be a straight subtraction or addition.

The interesting part is what the regexes don't include. This is not a general "normalize all East Asian width weirdness" function. It only targets visible ASCII-style characters: full-width ！ through ～, and half-width ! through ~. That means it intentionally leaves Chinese, Japanese, and Korean characters alone, which is the right safety tradeoff for a text cleaner.

But it also means a few things people might assume are covered are not. Ordinary spaces are excluded from the half-width regex, so converting "to full-width" does not turn spaces into full-width spaces. And the ideographic space U+3000 is outside this logic entirely. So the implementation is conservative, not comprehensive.

One other subtle detail: the detection count for width issues depends on the selected direction. In the computed stats, the tool counts either full-width characters that would be collapsed to ASCII, or ASCII characters that would be expanded to full-width. That's actually a good fit for the UI, because the count answers "how many characters will this mode change?" rather than some fuzzier question about how mixed the text feels.

The "AI artifact" cleanup is really a few blunt regex passes, applied in order

The part marketed as AI-copy-paste cleanup isn't a magical classifier. It's a small collection of explicit transforms:

const SMART_QUOTES_REGEX = /[“”‘’]/g;
const fixSmartQuotes = (str) =>
  str.replace(/[“”]/g, '"').replace(/[‘’]/g, "'");

const DASH_REGEX = /[—–]/g;
const fixDashes = (str) => str.replace(DASH_REGEX, "-");

const MARKDOWN_PATTERNS = [
  /\*\*(.+?)\*\*/g,
  /__(.+?)__/g,
  /`([^`]+)`/g,
  /^#{1,6}\s+/gm,
  /^[ \t]*[-*+]\s+/gm,
  /^>\s+/gm,
];

const fixMarkdown = (str) =>
  str
    .replace(/\*\*(.+?)\*\*/g, "$1")
    .replace(/__(.+?)__/g, "$1")
    .replace(/`([^`]+)`/g, "$1")
    .replace(/^#{1,6}\s+/gm, "")
    .replace(/^[ \t]*[-*+]\s+/gm, "")
    .replace(/^>\s+/gm, "");
Enter fullscreen mode Exit fullscreen mode

I actually prefer this to something smarter-sounding. The source is honest about what it does: curly quotes become straight quotes, em/en dashes become -, and a narrow set of Markdown markers get stripped while preserving the text they wrap.

After that, the component does a whitespace pass: repeated ASCII spaces collapse to one, runs of three or more newlines collapse to two, and every line gets trim() applied. The order matters, and the code tracks counts on the current text at each stage:

if (options.aiArtifacts) {
  if (options.smartQuotes) {
    aiApplied += countSmartQuotes(text);
    text = fixSmartQuotes(text);
  }
  if (options.dashes) {
    aiApplied += countDashes(text);
    text = fixDashes(text);
  }
  if (options.markdown) {
    aiApplied += countMarkdown(text);
    text = fixMarkdown(text);
  }
}

if (options.whitespace) {
  if (options.collapseSpaces) {
    wsApplied += countCollapseSpaces(text);
    text = fixCollapseSpaces(text);
  }
  if (options.collapseBlankLines) {
    wsApplied += countCollapseBlankLines(text);
    text = fixCollapseBlankLines(text);
  }
  if (options.trimLines) {
    wsApplied += countTrimLines(text);
    text = fixTrimLines(text);
  }
}
Enter fullscreen mode Exit fullscreen mode

That sounds minor, but I think it's the right design. The summary isn't just parroting the initial detection totals. It's reporting how many fixes each enabled stage actually applied after earlier stages had already mutated the text. So you don't get misleading double counts from a later pass operating on characters an earlier pass already removed.

Honest limitations and gotchas

The implementation is practical, but it definitely has edges.

The biggest one is the Markdown cleanup. It's intentionally lossy. If a line starts with #, >, or -, this tool assumes that marker is accidental formatting residue and strips it. Most of the time that's exactly what you want when pasting from a chat UI. But if your real content needs a leading hash, quote marker, or hyphen, this pass can be too aggressive.

It's also not a full Markdown parser, which is probably a good thing for a cleaner like this, but you can see the gaps. It handles **bold**, __bold__, inline backticks, ATX headings, unordered list markers, and blockquotes. It does not handle ordered lists, links, fenced code blocks, tables, or single-asterisk emphasis. So "Markdown cleanup" here really means "remove the handful of leftover markers I kept seeing in pasted text," not "round-trip arbitrary Markdown to plain text."

The width conversion has the same philosophy: safe and narrow. It won't touch CJK characters, which is good, but it also won't normalize ideographic spaces. If somebody expects this to solve every width-related typography issue in Unicode, it won't.

And the strangest detail in the whole component is probably the download behavior. The cleaner removes BOM characters from pasted input, but the TXT download function creates the blob with "" + outputText.value, which prepends a UTF-8 BOM back onto the saved file. I can see why — it makes the file open more reliably in some Windows apps — but it's worth knowing. If your definition of "clean" is "absolutely no U+FEFF anywhere," the copied output is cleaner than the downloaded file.

After running into these edge cases often enough, I turned this into a small free tool: Batch Text Format Cleaner. It runs entirely in the browser, which felt important for the kind of text people usually paste into something like this.

Available in other languages

Top comments (0)