DEV Community: Devendran M

Building an AI Tokenization Demo: From Workshop to App

Devendran M — Mon, 04 Aug 2025 11:33:53 +0000

How I turned learning from Steve Kinney's AI workshop into a hands-on web application to demystify tokenization

The Spark of Inspiration

Last week, I attended Steve Kinney's AI Fundamentals with Python Workshop on Frontend Masters. One concept that particularly fascinated me was tokenization – the process of converting human-readable text into tokens that AI models can understand, and vice versa.

I wanted to create something interactive that would help others (and future me) visualize this process. That's how the AI Tokenizer Demo was born.

What is Tokenization?

Before diving into the implementation, let's quickly understand what tokenization means in the AI context:

Tokenization is the process of breaking down text into smaller units called tokens. These tokens are then converted into numerical IDs that machine learning models can process. Different models use different tokenization strategies:

BERT: Uses WordPiece tokenization
GPT-2: Uses Byte Pair Encoding (BPE)
RoBERTa: Uses similar BPE approach as GPT-2
T5: Uses SentencePiece tokenization

The Project Vision

I wanted to create a simple, browser-based application that would:

Demonstrate tokenization visually - Show how text gets broken down into tokens
Support multiple models - Compare how different models tokenize the same text
Be educational - Help developers understand this fundamental AI concept
Require zero setup - Run entirely in the browser without API keys or backends

Technical Implementation

Choosing the Right Tools

Hugging Face's Transformers.js library.

This JavaScript library brings the power of transformer models directly to the browser without requiring a backend.

The Architecture

The app follows a simple, educational architecture:

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   User Input    │───▶│   Tokenization   │───▶│   Display       │
│   (Text/IDs)    │    │   Processing     │    │   Results       │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Key Features Implemented

Here are the core features:

Model Selection: The app supports four different tokenization models:

GPT-2 (max length: 1024 tokens)
BERT Base Uncased (max length: 512 tokens)
RoBERTa Base (max length: 512 tokens)
T5 Small (max length: 512 tokens)

Bidirectional Processing: Users can:

Tokenize & Encode: Enter text to see how it breaks down into tokens and numerical IDs
Decode: Enter token IDs to see the reconstructed text

Educational Display: The interface shows:

The selected model name
Maximum text length for the model
Tokenized representation
Encoded numerical IDs
Decoded text output

Development Insights

Learning from Real Implementation

Working with Transformers.js taught me several important lessons:

1. Browser-First AI is Powerful: The ability to run AI models directly in the browser without servers or API keys is game-changing. Users can experiment freely without worrying about rate limits or costs.

2. Model Loading is Asynchronous: When users select a different model, the app needs to download and initialize it. This taught me the importance of:

Loading indicators
Error handling
User feedback during async operations

3. Different Models, Different Behaviors: Each model tokenizes text differently. For example, the same input text might produce:

Different numbers of tokens
Different token boundaries
Different numerical representations

How to Run Locally

Want to explore the code and run it yourself? Here's how:

Quick Setup

# Clone the repository
git clone https://github.com/devendran-m/ai-tokenizer.git
cd ai-tokenizer

Running the Application

Since this is a pure HTML/CSS/JavaScript application that uses CDN resources, you just need to serve it over HTTP:

Option 1: Using Live Server (VS Code)

Install the Live Server extension
Right-click on index.html and select "Open with Live Server"

Option 2: Using Node.js serve

npx serve .

Navigate to http://localhost:5501 (or your server's port) and open index.html.

Project Structure

ai-tokenizer/
├── index.html          # Main HTML file with UI structure
├── script.js           # Core tokenization logic
├── styles.css          # Styling and layout
└── README.md          # Setup and usage instructions

Key Code Concepts

The application demonstrates several important concepts for developers:

Using Transformers.js via CDN:

<script type="module">
  import { AutoTokenizer } from 'https://cdn.jsdelivr.net/npm/@huggingface/transformers@latest/dist/transformers.min.js';
</script>

Model Loading Pattern:
The app loads different tokenizer models on demand, showing how to handle:

Asynchronous model loading
User feedback during loading states
Error handling for network requests

Tokenization Workflow:

User selects a model (GPT-2, BERT, RoBERTa, T5)
App loads the corresponding tokenizer
User inputs text to tokenize
App displays tokens and numerical IDs
User can decode IDs back to text

What You'll Learn

By exploring the code, you'll understand:

How different AI models tokenize text differently
Browser-based AI without backend requirements
Async/await patterns for model loading
DOM manipulation for dynamic UI updates
Error handling in AI applications

Example Usage

Step 1: Choose Your Model
The dropdown allows you to select from four different tokenization models. Each model will tokenize text differently, giving you insight into how various AI systems process language.

Step 2: Tokenize Text
Enter text and click "Tokenize & Encode". You'll see:

Tokenized: How the text breaks into individual tokens
Encoded: The numerical IDs representing each token
Model Info: Current model and its maximum text length

Step 3: Decode Tokens
Take the encoded IDs and paste them into the decode section to see how they convert back to text.

The Final Result

The completed application demonstrates tokenization with these key capabilities:

🔧 Model Flexibility: Choose between GPT-2, BERT, RoBERTa, and T5 tokenizers

📊 Visual Learning: See exactly how your text gets broken down into tokens

🔄 Bidirectional Processing: Convert text to tokens and tokens back to text

📋 Copy-Friendly Output: Easy to copy token IDs and decoded text for further experimentation

📱 No Installation Required: Runs entirely in the browser via CDN

🔗 Live Demo: https://ai-tokenizer.netlify.app/
🔗 Source Code: https://github.com/devendran-m/ai-tokenizer

Resources

Steve Kinney's AI Fundamentals Workshop - The original inspiration
Hugging Face Transformers.js Documentation - The library that powers the demo
Understanding Tokenization in NLP - Deep dive into tokenization concepts

Socials

LinkedIn | X

Building a Random Quotes Generator: A Step-by-Step Guide with Code

Devendran M — Tue, 21 Jan 2025 18:55:59 +0000

Creating a Random Quotes Generator App is a great way to learn coding by building something practical. This guide explains the steps I followed to create the app, including detailed code snippets to help beginners understand the process at the code level.

Project Overview

The app fetches random quotes using a public API, displays them on the screen, and allows users to copy or share the quotes. We’ll break down the steps to build this app and discuss the logic behind the code.

Step 1: Setting Up the HTML Structure

The first step is creating the HTML layout for the app. Here’s the basic structure:

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>Random Quotes Generator</title>
    <link rel="stylesheet" href="styles.css">    
</head>
<body>
    <div class="quote-generator">
        <h1 class="title">Random Quotes Generator</h1>
        <p class="subheading">"Randomness is not the absence of order, but the presence of possibilities."</p>
        <div class="quotebox">
            <h2 class="quote">Quote will appear here</h2>
            <h3 class="author">Author unknown</h3>
        </div>
        <div>
            <button class="quote-button" type="button" onclick="generateQuote()">New Quote</button>
        </div>
        <div>
            <p class="copyquote" onclick="copyQuote()"><img src="images/copy.png" class="icon-style" alt="Copy Icon"></p>            
            <p class="sharequote" onclick="shareQuote()"><img src="images/share.png" class="icon-style" alt="Share Icon"></p>
        </div>        
    </div>    
    <script src="./src/app/index.js"></script>
    <script src="./src/app/copyQuote.js"></script>
    <script src="./src/app/shareQuote.js"></script>
</body>
<footer>
    <div class="footer">
        <p class="footer-left">
            <a class="credits" href="https://github.com/theriturajps/Quotes-API" target="_blank" rel="noopener">Credit: Quotes API</a>
        </p>
        <p class="footer-right">
            <a href="https://github.com/theriturajps/Quotes-API" class="menu" target="_blank" rel="noopener"><img src="images/dev_github.png" class="icon-style" alt="GitHub Icon"></a>
            <a href="https://github.com/theriturajps/Quotes-API" class="menu" target="_blank" rel="noopener"><img src="images/dev_x_twitter.png" class="icon-style" alt="Twitter Icon"></a>
            <a href="https://github.com/theriturajps/Quotes-API" class="menu" target="_blank" rel="noopener"><img src="images/dev_linkedin.png" class="icon-style" alt="LinkedIn Icon"></a>
        </p>
        <h5>Built with &hearts; from Dev </h5>
    </div>
</footer>
</html>

This code defines the structure of the app, including elements to display the quote, buttons to fetch new quotes, and icons for copy and share functionality.

Step 2: Handling CORS Issues with a Proxy Server

To fetch quotes from an external API, you need to handle CORS (Cross-Origin Resource Sharing). We achieve this using a simple Express.js proxy server.

proxy.js:

const express = require("express");
const fetch = require("node-fetch");
const cors = require("cors");
const app = express();

// Enable CORS
app.use(cors());

// Define a route for proxying API requests
app.get("/api/quote", async (req, res) => {
    try {
        const response = await fetch("https://qapi.vercel.app/api/random");
        const data = await response.json();
        res.json(data);
    } catch (error) {
        res.status(500).json({ error: "Failed to fetch data from API" });
    }
});

const PORT = 4000;
app.listen(PORT, () => console.log(`Proxy server running on http://localhost:${PORT}`));

This acts as a local proxy to fetch quotes from the API and bypass CORS restrictions.

Step 3: Fetching Quotes with JavaScript

The app fetches a new quote when the "New Quote" button is clicked. Here’s how the logic works:

index.js:

const quote = document.querySelector(".quote");
const author = document.querySelector(".author");

async function generateQuote() {
    const url = 'http://localhost:4000/api/quote';

    try {
        const response = await fetch(url);
        if (!response.ok) {
            throw new Error(`HTTP error! Status: ${response.status}`);
        }
        const result = await response.json();
        quote.textContent = result.quote || "No quote available.";
        author.textContent = result.author || "Author Unknown";
    } catch (error) {
        console.error("Error fetching quote:", error);
        quote.textContent = "Failed to fetch quote. Try again later.";
    }
}

This script fetches data from the proxy server and updates the app’s UI with the fetched quote and author.

Step 4: Adding the Copy Feature

To enable users to copy quotes, we use the Clipboard API.

copyQuote.js:

async function copyQuote() {
    try {
        let quote = document.querySelector(".quote").textContent;
        let author = document.querySelector(".author").textContent;

        await navigator.clipboard.writeText(`${quote} - ${author}`);
        alert("Content copied to clipboard");
    } catch (error) {
        console.error('Failed to copy:', error);
    }
}

When the user clicks the copy icon, the quote and author are copied to the system clipboard.

Step 5: Adding the Share Feature

The Navigator API allows users to share quotes on WhatsApp.

shareQuote.js:

async function shareQuote() {
    let quote = document.querySelector(".quote").textContent;
    let author = document.querySelector(".author").textContent;

    const quoteContent = `${quote} - ${author}`;
    const message = encodeURIComponent(quoteContent);

    if (window.screen.width < 768 || navigator.userAgent.match(/Android|iOS/)) {
        window.open(`whatsapp://send?text=${message}`, '_blank');
    } else {
        window.open(`https://web.whatsapp.com/send?text=${message}`, '_blank');
    }
}

This script adapts sharing functionality for both mobile and desktop devices.

Step 6: Styling the App

Using CSS, I styled the app to make it visually appealing and responsive. Key aspects include centering elements and designing for various screen sizes.

html, body {
    background: rgb(171,129,205);
    background: linear-gradient(0deg, rgba(171,129,205,1) 0%, rgba(34,42,104,1) 100%);         
    margin: 0;
    padding: 0;
    height: 100%;    
}

main {
    flex: 1; /* Fills available space */
    display: flex;
    justify-content: center;
    align-items: center;
    flex-direction: column;
}

.title, .subheading {
    align-items: center;
    text-align: center;
    color: #fafffd
}

.subheading {
    align-items: center;
    text-align: center;
    color: #fa824c
}

p {
    text-align: center; 

}

.quote-generator, .quote-box, .footer {
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    margin: 100 auto;
    margin-top: 50px;
}

.quotebox   {
    width: 300px auto;    
    height: 200px auto;    
    margin-top: 100px;
    padding: 10px;
    border: 2px solid #9fd356;
    overflow: auto;
    border-radius: 20px;    
}

.quote  {
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: center;
    margin: 100 auto;
    color: #fafffd;        
}

.quote-button {
    padding: 15px;
    margin: 15px;
    border-radius: 5px;
    background-color: #e2adf2;
    color: black;
    font-size: larger;    
}

.copyquote, .sharequote {
    display: flex;
}

.author {
    color: #9fd356;;    
}

.footer {
    display: flex;
    flex-direction: column;
    align-items: center;
    justify-content: end;
    height: 200px;
    margin: 200px;    
}

.footer-left {
    font-size: 20px;    
    float: center;    
}



.footer-right {
  margin-top:20px;
  float:center;  
}

a.menu:hover {
    border-radius: 2px;
    font-size: 20px;    
}

.icon-style {
    height:60px;
    margin-left:20px;
    margin-top:5px;
}

.icon-style:hover {
    background-color: #ab81cd;
    padding:5px;
    border-radius: 15px;
    transition: 0.5s;
    transition-duration: 0.25s;
    color: azure;
}

Step 7: Launching the App

The app can be built from the example code. Alternately, the subsequent steps may be followed to attempt the application.

To launch and use the app, follow these steps:

Clone the Repository:

   git clone https://github.com/yourusername/random-quotes-generator.git
   cd random-quotes-generator

Install Dependencies: Make sure you have Node.js installed on your system. Then run:

   npm install

Start the Proxy Server: Run the following command to start the proxy server:

   node proxy.js

The proxy server will run on http://localhost:4000.

Open the App: Open the index.html file in your browser to interact with the app. Use the "New Quote" button to fetch quotes, copy them using the copy icon, or share them via WhatsApp using the share icon.

Project Structure:

index.html: Main UI for the app.
proxy.js: Proxy server to handle CORS issues.
copyQuote.js: Handles copying quotes to the clipboard.
shareQuote.js: Handles sharing quotes via WhatsApp.
index.js: Core functionality for fetching and displaying quotes.

API Credit:

Quotes provided by Quotes API.

Conclusion

This guide demonstrates how to build a Random Quotes Generator app, covering essential steps like fetching API data, handling CORS, and implementing copy and share features. It can help:

Understand API integration and CORS issues.
Explore browser APIs like Clipboard and Navigator.
Practice JavaScript fundamentals.

Happy coding! I am a learner and would love to hear feedback from the community and experienced developers. Your suggestions are valuable to me, and I am eager to learn and improve. You are welcome to reach out!

GitHub | LinkedIn | X