When copying generated text, documentation, or code snippets from ChatGPT, Claude, or Gemini into a CMS, Notion, Gmail, or code editor, two recurring issues come up for developers and technical writers:
-
Formatting & Invisible Unicode Artifacts: Raw
**bold**asterisks,###heading hashes, overused AI em-dashes (—), broken PDF line wraps, and invisible zero-width characters (U+200B,U+FEFF,U+00A0) that silently break string matching, regex validation, and character limits. - Context Window & Token Estimation: Knowing how many Byte-Pair Encoding (BPE) tokens a long prompt, JSON payload, or multi-language document consumes before sending it to GPT-4o (128K), Claude 3.5 Sonnet (200K), or Gemini 1.5 Pro (1M)—without uploading confidential text to a third-party server.
To solve both problems with zero server round-trips, I built two new 100% client-side utilities as part of TextScope.
1. Stripping Markdown, Em-Dashes, and Hidden Unicode Characters
Hidden Unicode characters often sneak into copied text from rich-text editors, PDFs, and AI chat interfaces:
| Unicode Point | Character Name | Common Issue in Production |
|---|---|---|
U+200B |
Zero-Width Space | Breaks exact string comparisons, SQL queries, and JSON keys |
U+FEFF |
Byte Order Mark (BOM) | Causes unexpected syntax errors at the start of scripts or CSVs |
U+00A0 |
Non-Breaking Space | Prevents standard CSS word wrapping and breaks whitespace regex |
U+2014 |
Em-Dash (—) |
Overused by LLMs; makes technical and editorial copy look robotic |
With the AI Text Cleaner & Markdown Stripper, you can paste any LLM or PDF output and toggle exactly what to clean locally in your browser:
- Strip Markdown syntax (
**bold**,*italic*,###headings, code fences, and inline links) while keeping clean paragraph breaks. - Detect and remove invisible zero-width characters (
U+200B,U+200C,U+200D,U+FEFF,U+00A0) with an exact count of removed artifacts. - Convert AI em-dashes (
—) and curly quotes (“”) into standard ASCII punctuation. - Unwrap hard single line breaks copied from multi-column PDFs without merging double-newline paragraphs.
2. Estimating LLM Prompt Tokens & Context Window Headroom
In standard English prose, 1 token ≈ 0.75 words (or roughly 4 characters), meaning 1,000 words is ~1,330 tokens. However, source code, JSON keys, Markdown tables, and CJK scripts (Japanese, Korean, Chinese) have a higher token-to-character ratio (1.2–1.8 tokens per character).
The LLM Prompt Token Counter & Words-to-Tokens Calculator runs entirely in vanilla JavaScript and calculates:
- Estimated BPE token count, total words, character count, and token-to-word ratio.
- Real-time context window utilization across 8K, 32K, 128K (GPT-4o / Llama 3.1), 200K (Claude 3.5 Sonnet), and 1M (Gemini 1.5 Pro) limits.
- Single-spaced page equivalents for long-form technical docs and books.
Full Suite of Client-Side Writing & SEO Utilities
All tools in the suite run 100% locally in the browser (no sign-up, no server uploads) and support 6 languages (English, Spanish, German, Portuguese, Japanese, and Korean):
- Word Counter & Flesch Readability Analyzer – Real-time word/character count, Flesch Reading Ease score, Flesch-Kincaid Grade Level, and 1–3 word n-gram phrase density.
- AI Text Cleaner & Format Stripper – One-click removal of ChatGPT/Claude Markdown, em-dashes, zero-width spaces, and broken PDF line breaks.
- LLM Token Counter & Context Calculator – Instant words-to-tokens conversion and context window percentages.
-
SEO Headline Analyzer & Google SERP Preview – Title tag pixel width (
600px) and character length (50–60 chars) checker with live search snippet preview. -
Reading & Speaking Time Calculator – Converts word counts into silent reading (
200 WPM) and speech/presentation durations (130 WPM). -
Case Converter & URL Slug Generator – Converts text between APA Title Case, Sentence case,
camelCase,snake_case, and clean SEO URL slugs.
You can also check out the full multilingual documentation and reference benchmarks in the public GitHub repository: GS-Seven/textscopes on GitHub.
Feedback and feature suggestions are very welcome!
Top comments (0)