Building a Base64 Tool That Actually Handles UTF-8 (Without Losing Your Mind)

The Problem That Started It All

I was working on a project that required sending configuration data between different services. The payload contained Chinese characters, emojis, and the occasional accented character from a French colleague's name in the comments. Classic stuff.

The first time I tested it, everything worked. The second time, I got back a string of é and ’ instead of actual text. You know the drill — mojibake. The kind of thing that makes you question every life decision that led you to this moment.

The root cause? I was using btoa() and atob() directly on strings containing non-ASCII characters. These functions only handle Latin-1 (ISO-8859-1) natively. Feed them a Chinese character and they'll happily mangle it into something that looks like it fell out of a corrupted file.

Why Not Just Use an Existing Tool?

I could have opened any of the dozens of online Base64 converters. But I had specific requirements:

• I needed something that handles UTF-8 correctly (the mojibake problem above)
• I wanted it to work offline
• I didn't want to paste sensitive config data into some random website
• I was tired of tools that required clicking a "Convert" button when I just wanted live feedback

So I decided to build a small browser-based tool. Because apparently I enjoy reinventing wheels.

The UTF-8 Problem (The Part That Actually Matters)

Here's the core issue with btoa() and atob():

// This works fine
btoa("hello world");

// This throws an error or produces garbage
btoa("你好世界");

The fix is straightforward but easy to miss if you're in a hurry:

function encodeUTF8(str) {
    return btoa(String.fromCharCode(...new TextEncoder().encode(str)));
}

function decodeUTF8(base64) {
    return new TextDecoder().decode(
        Uint8Array.from(atob(base64), c => c.charCodeAt(0))
    );
}

The TextEncoder converts the string to UTF-8 bytes, and String.fromCharCode(...) turns those bytes into a string that btoa() can handle. The decode path reverses this process.

One gotcha: for very large inputs, String.fromCharCode(...array) can blow the call stack. In production, you'd want a chunked approach. But for typical text inputs, this works fine.

The Real-Time Conversion Decision

The most interesting design decision was whether to convert on every keystroke or wait for user action.

Option A: Real-time conversion

• Pro: Instant feedback, feels magical
• Con: Can be janky with large inputs, potentially wasteful

Option B: Manual conversion button

• Pro: Predictable, performant
• Con: Extra click, feels dated

I ended up going with a hybrid: real-time by default, but with a manual override. Users can toggle it off if they're pasting a massive file. The implementation is simple — just listen to the input event and debounce:

let debounceTimer;
input.addEventListener('input', () => {
    if (!realTimeToggle.checked) return;
    clearTimeout(debounceTimer);
    debounceTimer = setTimeout(convert, 150);
});

The debounce is critical. Without it, every keystroke triggers a full conversion cycle, which causes noticeable lag on longer inputs.

The Encoding/Decoding Direction Problem

Here's something I didn't think about until I started building: when you swap between encode and decode modes, what happens to the existing input?

If someone types "hello" in encode mode, then switches to decode mode, should "hello" be decoded as Base64? That would produce garbage. The sensible behavior is to clear the input when the mode changes.

But wait — there's a more interesting UX pattern. What if the user wants to decode something they just encoded? They'd have to copy the output, switch modes, and paste it back. That's annoying.

This is where the swap button comes in. Instead of manually copying and switching, one click moves the current output into the input field and flips the mode. It's a small touch, but it makes the tool feel much more fluid.

The Error Handling Trap

Decoding invalid Base64 is a surprisingly common failure mode. Users paste in text that looks like Base64 but isn't — maybe it has a space in it, or they copied a URL fragment by accident.

The naive approach:

try {
    const result = atob(input);
    // ...
} catch (e) {
    showError("Invalid Base64");
}

This works, but it's not very helpful. The better approach is to validate the input format first and give specific feedback:

function isValidBase64(str) {
    if (str.length % 4 !== 0) return false;
    return /^[A-Za-z0-9+/]*={0,2}$/.test(str);
}

This catches the most common issues: wrong length and invalid characters. The error message can then tell the user what's wrong rather than just "that didn't work."

What the AI Got Right and Wrong

I built this tool with AI assistance, and I want to be honest about the experience.

What the AI nailed:

• The initial structure and layout
• The i18n setup with language detection
• The dark mode implementation using CSS variables
• The basic encode/decode logic

Where I had to step in:

The first version of the UTF-8 handling was wrong. The AI used encodeURIComponent as a workaround, which produces a URL-encoded string that isn't proper Base64. It worked for basic cases but broke with certain character combinations.

I had to explain the issue and iterate:

"The current implementation uses encodeURIComponent which produces incorrect Base64. We need to use TextEncoder/TextDecoder instead. Here's the correct approach..."

The AI then generated the correct implementation. But here's the thing — I only knew it was wrong because I'd hit this exact bug before. If I hadn't, I would have shipped broken code.

The lesson: AI is great at scaffolding and boilerplate, but you still need to understand the domain well enough to catch subtle bugs.

The i18n Decision

I'm building this tool for a global audience, so I needed English and Chinese support. The requirement was simple: detect the browser language, allow a URL override, and default to Chinese.

The implementation was straightforward:

const translations = {
    'zh': {
        'title': 'Base64 编码/解码',
        'encode': '编码',
        'decode': '解码',
        // ...
    },
    'en': {
        'title': 'Base64 Encode/Decode',
        'encode': 'Encode',
        'decode': 'Decode',
        // ...
    }
};

function detectLanguage() {
    const urlLang = new URLSearchParams(window.location.search).get('lang');
    if (urlLang === 'en' || urlLang === 'zh') return urlLang;
    return navigator.language.startsWith('zh') ? 'zh' : 'en';
}

The t('key') function then looks up translations:

function t(key) {
    return translations[currentLang][key] || translations['zh'][key] || key;
}

Simple, but it works. No external dependencies, no build step, just a plain object.

Performance Considerations

For a tool like this, performance is mostly about not being stupid. The main concerns:

1. Large inputs: A 10MB file will freeze the browser if you try to process it synchronously on every keystroke. The debounce helps, but you might also need to cap the input size.

2. String operations: String.fromCharCode(...array) is fine for typical inputs but will throw a RangeError for arrays larger than about 65K elements. For robustness, you'd chunk it:
   

function bytesToBinaryString(bytes) {
    let result = '';
    const chunkSize = 0x8000;
    for (let i = 0; i < bytes.length; i += chunkSize) {
        result += String.fromCharCode.apply(null, bytes.subarray(i, i + chunkSize));
    }
    return result;
}

1. DOM updates: Don't update the output textarea on every keystroke. Batch updates or use requestAnimationFrame if needed.

What I Learned

The "it works on my machine" moment: The first version of the tool worked perfectly with English text. It wasn't until I tested with Chinese characters that the encoding bug surfaced. This is a reminder to always test with the character sets your users will actually use.

The "it's always a CSS issue" moment: The dark mode implementation initially had a bug where textareas kept a light background in dark mode. It was a specificity issue — the background property on textarea was overriding the CSS variable. The fix was to use the variable consistently:

textarea {
    background: var(--bg-input);
    color: var(--text);
}

The "AI is a junior developer" moment: AI-assisted development is powerful, but it's not a replacement for understanding the fundamentals. The AI could write the code, but I had to know enough to catch the subtle bugs. It's like having a very fast junior dev who occasionally makes confident mistakes.

The Result

During this process, I built a small browser-based tool to make this workflow easier. It handles UTF-8 correctly, supports real-time conversion, and works entirely in the browser — no data ever leaves your machine.

The full implementation is a single HTML file with inline CSS and JavaScript, which makes it easy to deploy or even save locally for offline use.

Final Thoughts

Base64 encoding seems simple until it isn't. The UTF-8 handling is the kind of thing that looks trivial in theory but causes real production issues. If you're building anything that processes text, take five minutes to test with non-ASCII characters. Your future self will thank you.

And if you're using AI to write code, remember: it's a tool, not a replacement for understanding. The best workflow is to use AI for the parts that are tedious and mechanical, then apply your own knowledge to catch the subtle issues.

---

You can try the tool at Craftvo if you're curious about the implementation.