DEV Community: Aivan Carlos Tuquero

Breaking Down Tokenization in LLMs: How AI read your words

Aivan Carlos Tuquero — Fri, 29 Aug 2025 02:42:50 +0000

When you interact with Large Language Models (LLMs) like GPT-5, LLaMA, or Claude, it feels like you’re sending sentences and paragraphs. But under the hood, the model doesn’t “see” text the way we do.

Instead, everything you type is broken down into tokens—tiny units that sit at the heart of how LLMs process, generate, and price their outputs.

In this post, we’ll dive deep into tokenization in machine learning, why it matters for developers, and how you can actually experiment with tokenizers using tools like tiktokenizer.

What Is Tokenization?

Tokenization is the process of splitting text into smaller units called tokens, which are then mapped to numerical IDs and embedded into vectors for processing.

For us, words are natural chunks of meaning.
For machines, tokens are the bridge between raw text and mathematical computation.

Example (GPT-style tokenizer):

Text: "Machine learning is amazing!"
Tokens: ["Machine", " learning", " is", " amazing", "!"]

Each token maps to an integer ID:

[1234, 5678, 90, 4321, 999]

Those integers are what the model actually processes.

Why Tokenization Matters in LLMs

Tokenization isn’t just a preprocessing detail—it has major real-world implications:

Context Window Limitations Models can only handle a certain number of tokens in memory (e.g., 4k, 32k, or even 1M tokens).

Your 10,000-word novel draft might not fit because tokenization can expand the input.

Pricing & API Costs Most LLM providers charge per token, not per word.

A “short” email could be 60 words but 90+ tokens depending on how it’s split.
Optimizing your prompts can save serious money.

Language & Domain Efficiency Tokenization efficiency differs by language and domain:

English words often tokenize cleanly.
Agglutinative languages (like Turkish or Finnish) might explode into many tokens.
Programming code tends to be token-heavy (function() {} often becomes multiple tokens).

Tokenization Techniques in LLMs

Let’s explore the main strategies used in modern NLP:

1. Word-Based Tokenization (Old School)

Splits on spaces/punctuation.

Example: "Machine learning rocks" → ["Machine", "learning", "rocks"].
Problem: Vocabulary explosion (“running”, “runs”, “ran” are all separate).

2. Subword Tokenization (BPE, WordPiece, Unigram LM)

Breaks text into subwords, balancing vocabulary size and generalization.

Example:

  "unhappiness" → ["un", "happi", "ness"]

Used in BERT and early Transformer models.

3. Byte-Pair Encoding (BPE, GPT-2/GPT-3)

Starts at character level, merges frequently co-occurring pairs.

Handles rare words better.
Example: "programming" → ["program", "ming"].

4. Byte-Level Tokenization (GPT-2 and beyond)

Works at the raw byte level (UTF-8).

Advantages: Handles emojis, accented characters, and rare text seamlessly.
Example: "🐱" → ["🐱"] instead of breaking into unknown tokens.

5. Character-Level Tokenization (Rare for LLMs)

Splits into every character.

Example: "AI" → ["A", "I"].
Downsides: Very long sequences → inefficient for LLMs.

Comparing Tokenizers on the Same Text

Let’s take the sentence:

I love programming in Python 🐍

Word-based: ["I", "love", "programming", "in", "Python", "🐍"]
BPE (GPT-2): ["I", " love", " program", "ming", " in", " Python", " 🐍"]
Character-level: ["I", " ", "l", "o", "v", "e", " ", "p", "r", ...]

Notice how BPE splits “programming” into “program” + “ming,” while emojis remain intact under byte-level tokenization.

Now this is what it looks like using the popular cl100k_base

Why Developers Should Care About Tokenization

As a developer building on top of LLMs, tokenization directly affects your work:

API Usage & Billing – Every token counts toward input/output cost.
Prompt Design – Understanding tokens helps you craft concise, efficient prompts.
Language Support – Tokenization efficiency varies across languages; testing is essential.
Optimization – Preprocessing your text (Compressing JSON, trimming whitespace) can reduce token usage by 10–20%.

So, run your prompts through a tokenizer first. It’ll save you money, prevent cutoff issues, and give you a clearer picture of how your app interacts with LLMs.

How to Kill Vulnerabilities in Your Node.js App: A Guide to Writing Secure JavaScript Code

Aivan Carlos Tuquero — Sun, 10 Nov 2024 15:33:26 +0000

Js/Ts and Node.js have revolutionized the software engineering world, but with great power comes great responsibility🕷️. With so many packages and the fast pace of engineering, vulnerabilities are bound to sneak in. In this article, we’ll tackle the most common vulnerabilities lurking in the JavaScript ecosystem and show you exactly how to “kill” them with secure code practices.

1. Dependency Vulnerabilities

Problem: The JavaScript ecosystem relies heavily on packages from places like npm. When you install these packages, you often pull in additional dependencies. Unfortunately, not all packages are maintained with security in mind, and some even come with malicious code intentionally.

Solution:

Audit Dependencies: Run npm audit to check for any known vulnerabilities in your dependencies. This will give you a report and suggestions for fixing issues.
Regularly Update Packages: Use npm outdated to check for outdated packages, especially those with security patches. Staying up-to-date helps prevent vulnerabilities.
Use Security-Focused Tools: Tools like Snyk or OWASP Dependency-Check scan your dependencies for known vulnerabilities.

2. Insecure Configurations

Problem: Leaving default configurations, especially in production, can expose your application to attackers. Things like enabling verbose logging, leaving debug modes on, or allowing CORS for all origins can create security holes.

Solution:

Environment-Specific Configurations: Set different configurations for development and production. For example, disable debug mode and reduce logging in production.
Environment Variables: Keep sensitive information (e.g., database credentials, API keys) in environment variables and use libraries like dotenv to manage them securely.
Use a .env File for Sensitive Data: Never store credentials or sensitive data in your codebase. Use a .env file, and don’t forget to add it to .gitignore.

3. Injection Attacks (SQL/NoSQL Injection)

Problem: Injection attacks happen when user input is incorrectly processed and treated as executable code or a database command. For instance, SQL injection can allow attackers to manipulate database queries and access sensitive data.

Solution:

Parameterized Queries: Always use parameterized or prepared statements when interacting with databases. Libraries like pg for PostgreSQL or mongoose for MongoDB support these secure methods.
Sanitize User Input: Use a library like validator to sanitize input, especially when dealing with forms or other sources of user input.

4. Cross-Site Scripting (XSS)

Problem: XSS attacks happen when attackers inject malicious scripts into your application. For example, if your app displays user-generated content without sanitizing it, an attacker could inject JavaScript that gets executed by other users’ browsers.

Solution:

Escape User Input: Make sure that any user-generated content displayed on the front end is escaped. This way, it’s treated as plain text and not as executable code.
Content Security Policy (CSP): A CSP allows you to define which scripts, images, and styles are allowed to run on your website. It’s a powerful way to limit XSS attacks. You can set up a CSP header in your server configuration.
Use a Trusted Library: If you use templating libraries (e.g., Handlebars, EJS), they often have built-in escaping features. Don’t disable them.

5. Cross-Site Request Forgery (CSRF)

Problem: CSRF attacks trick users into executing unwanted actions on a different website where they’re authenticated. For example, a user logged into their bank account could unknowingly transfer money to another account by clicking a link in a malicious email.

Solution:

Use CSRF Tokens: When using forms, implement CSRF tokens to verify that each request is legitimate. Many frameworks, such as Express, have middleware like csurf to help prevent CSRF attacks.
Double Submit Cookies: This is another approach where you set a unique token in a cookie and require the same token to be submitted in the request payload.

const express = require('express');
const csurf = require('csurf');
const cookieParser = require('cookie-parser');

const app = express();

// Use cookie parser and csrf middleware
app.use(cookieParser());
app.use(csurf({ cookie: true }));

// Middleware to add CSRF token to all responses
app.use((req, res, next) => {
  res.locals.csrfToken = req.csrfToken();
  next();
});

app.get('/form', (req, res) => {
  // Render a form with the CSRF token
  res.send(`
    <form action="/submit" method="POST">
      <input type="hidden" name="_csrf" value="${res.locals.csrfToken}">
      <input type="text" name="data">
      <button type="submit">Submit</button>
    </form>
  `);
});

app.post('/submit', (req, res) => {
  res.send('Data received securely!');
});

app.listen(3000, () => console.log('Server running on http://localhost:3000'));

6. Insecure Data Storage

Problem: Storing sensitive data, like passwords or personal information, without encryption or secure storage methods can make it easy for attackers to steal this data if they gain access.

Solution:

Encrypt Sensitive Data: Use encryption for sensitive data at rest. For example, use libraries like bcrypt for hashing passwords.
Use HTTPS for Data in Transit: Encrypt data in transit by forcing HTTPS connections. Services like Let’s Encrypt offer free SSL certificates to secure your application.

const bcrypt = require('bcrypt');

async function registerUser(password) {
  // Hash the password with a salt round of 10
  const hashedPassword = await bcrypt.hash(password, 10);
  // Save hashedPassword to the database instead of the plain password
  console.log('Hashed Password:', hashedPassword);
}

async function loginUser(enteredPassword, storedHashedPassword) {
  // Compare the entered password with the stored hashed password
  const match = await bcrypt.compare(enteredPassword, storedHashedPassword);
  if (match) {
    console.log('Login successful!');
  } else {
    console.log('Invalid password');
  }
}

// Example usage
registerUser('gokuIsStrong');

7. Server-Side Vulnerabilities

Problem: Since Node.js runs on the server, any unhandled errors or improperly configured server settings can lead to security issues.

Solution:

Error Handling: Always handle errors gracefully, and avoid exposing sensitive information in error messages. Instead of sending detailed error messages, use generic messages in production.
Limit Request Size: Large payloads can overload your server, so limit the request body size using middleware like body-parser to prevent attacks like denial of service (DoS).
Run as Non-Root User: Avoid running your application with root privileges. In the event of a compromise, this can help limit the damage an attacker can do.

Final Tips

Securing your Node.js and JavaScript applications takes time, but it’s a necessary investment. Here are some final quick tips:

Use HTTPS Everywhere: Encrypting data in transit is critical to protect user data.
Avoid eval() and similar functions: Functions like eval() can execute arbitrary code, making your app vulnerable to injection attacks. Avoid using them whenever possible.
Keep Dependencies to a Minimum: Only install packages you really need. Each package introduces a potential vulnerability, so be selective.

By following these tips and regularly updating your knowledge on security practices, you’ll be better equipped to keep your Node.js applications safe. Security is an ongoing process, but with a proactive approach, you can significantly reduce your application’s vulnerability footprint.

Conclusion

As much as we aim to secure our code, the truth is there's no such thing as a perfectly secure application, and we can’t kill every vulnerability. New vulnerabilities are discovered regularly, libraries are updated, and our code constantly evolves. Security is an ongoing process that requires vigilance, consistent updates, and a proactive reverse engineering mindset.

Here are a few additional ways to keep improving your code security:

Regularly Review Code: Conduct code reviews with a focus on security. Peer reviews help catch vulnerabilities that one developer might miss.
Automate Security Testing: Use CI/CD pipelines with automated security testing to catch issues early in development. Tools like GitHub Dependabot, Snyk, and npm audit can be integrated into your workflow for continuous scanning.
Stay Informed: Security threats evolve, so stay updated with security news, libraries, and practices. Following communities like OWASP and the Node.js security team can keep you informed of the latest threats and mitigation techniques.
Least Privilege Principle: Limit permissions and access levels in your app and database. The principle of least privilege ensures that each part of your application only has the access it absolutely needs to function, reducing potential damage from a breach.
User Education: Encourage safe practices, especially for teams with access to sensitive code and environments. Developer security training can help avoid common mistakes that lead to vulnerabilities.

By being vigilant and continuously learning, you’ll be better equipped to manage security risks in the fast-paced Node.js ecosystem. Remember: it’s about reducing risk, not achieving perfection. Happy coding, and here’s to safer, more secure applications!

Why JavaScript Is Not 'True' OOP

Aivan Carlos Tuquero — Sun, 11 Aug 2024 09:27:07 +0000

JavaScript is a language loved by many, but it often gets a bit of a bad rap when it comes to object-oriented programming (OOP). If you’ve come from a background in languages like Java, C++, or C#, you might have heard that JavaScript isn’t a "true" OOP language. But what does that really mean? Let’s unpack this concept and understand why JavaScript stands apart from traditional OOP languages.

1. Prototype-Based Inheritance: The Core Difference

At the heart of the matter is JavaScript’s use of prototype-based inheritance. In traditional OOP languages, you create classes, which are blueprints for objects. These classes can inherit properties and methods from other classes, forming a clear hierarchy.

JavaScript, on the other hand, doesn’t use classes in the same way (at least, not until ES6—and even then, it’s more syntactic sugar, but more on that later). Instead, JavaScript uses prototypes. Every object in JavaScript has a prototype, and objects can inherit directly from other objects. This means there’s no class-to-object translation like in classical OOP languages. Instead, objects beget other objects.

const animal = {
  speak() {
    console.log("Animal speaks");
  }
};


const dog = Object.create(animal);
dog.speak(); // Output: Animal speaks
//Here, dog inherits directly from animal without any class in sight.

2. Encapsulation? More Like Pseudo-Encapsulation
Encapsulation is one of the pillars of OOP. It refers to the bundling of data (attributes) and methods that operate on that data within a single unit or class, with the ability to hide the internal workings from the outside world.

JavaScript’s traditional lack of strict encapsulation is another reason it’s not considered "true" OOP. In the earlier days of JavaScript, there wasn’t an official way to create private variables—everything was public and could be accessed or modified.

With the introduction of ES6, JavaScript added class syntax and, more recently, private fields using a # prefix, which has helped with encapsulation. However, these features are relatively new and are not as robust or enforced as in other OOP languages.

class Animal {
  #name;

  constructor(name) {
    this.#name = name;
  }

  getName() {
    return this.#name;
  }
}

const dog = new Animal("Buddy");
console.log(dog.getName()); // Output: Buddy
console.log(dog.#name); // SyntaxError: Private field '#name' must be declared in an enclosing class

//This private field syntax helps, but the flexibility of JavaScript means it’s still easy to break encapsulation in other ways.

3. Functions Are First-Class Citizens
In JavaScript, functions are first-class citizens. This means you can treat functions like any other object: assign them to variables, pass them as arguments to other functions, and even return them from functions.

While this is a powerful feature, it’s another reason JavaScript doesn’t fit the traditional OOP mold. In many OOP languages, methods are tightly coupled with classes and objects, whereas JavaScript’s functions exist independently and can be used in more dynamic ways.

function sayHello() {
  return function() {
    console.log("Hello!");
  };
}

const greet = sayHello();
greet(); // Output: Hello!

This flexibility is fantastic for functional programming but diverges from the strict method-object coupling seen in classic OOP.

4. Dynamic Typing: A Blessing and a Curse
JavaScript is dynamically typed, meaning variables can hold any type of value and their type can change at runtime. This dynamic nature is at odds with the strict type systems found in most OOP languages, where types are explicitly declared and enforced.

Dynamic typing allows for rapid development and flexibility, but it can also lead to unpredictable behavior and bugs that are hard to track down—something that strict OOP languages attempt to mitigate with their rigid type systems.

let x = "Hello";
x = 42; // No problem in JavaScript!

While dynamic typing is powerful, it’s one more reason why JavaScript doesn’t fit neatly into the OOP paradigm.

5. The class Syntax: Just Syntactic Sugar
In ES6, JavaScript introduced the class keyword, which brought a more familiar syntax to those coming from traditional OOP languages. However, it’s important to understand that this is mostly syntactic sugar. Under the hood, JavaScript is still using its prototype-based inheritance system.

class Animal {
  constructor(name) {
    this.name = name;
  }

  speak() {
    console.log(`${this.name} speaks`);
  }
}

class Dog extends Animal {
  speak() {
    console.log(`${this.name} barks`);
  }
}

const dog = new Dog("Buddy");
dog.speak(); // Output: Buddy barks

While this looks like class-based inheritance, it’s important to remember that JavaScript is still doing its own thing under the hood with prototypes. This can lead to confusion for developers who expect class-based behavior.

Conclusion: IS JS TRUELY OOP?
So, is JavaScript truly object-oriented? The answer is both yes and no. JavaScript is incredibly flexible and can be used in an object-oriented way, but it doesn’t adhere to the strict rules and paradigms of traditional OOP languages. Instead, JavaScript offers its own unique blend of prototypes, dynamic typing, and functional programming capabilities, making it a powerful and versatile language—just not a "true" OOP one in the classical sense.

Whether you see this as a strength or a limitation largely depends on your background and the kind of programming you’re used to. But one thing is clear: JavaScript’s approach to OOP is unique, and understanding its nuances will make you a better developer.

From Toy Blocks to Code Blocks: My Journey into Tech

Aivan Carlos Tuquero — Sat, 06 Apr 2024 15:55:34 +0000

Young me 🪀

Growing up in the Philippines, I was fortunate to have a family who valued education and could afford to buy me toys. Among those toys, I always found myself drawn to the ones that offered complexity and challenge – Lego, Transformers, and building blocks. Every time I got my hands on a set, I was captivated by the idea of transforming simple pieces into intricate creations like helicopters and cars.

This, without me even realizing it, ignited my passion for engineering.

My love for puzzles extended beyond Legos. Anything that required logical thinking and a bit of brainpower to solve had me hooked. There was a certain satisfaction in untangling a seemingly impossible mess and finding the solution.

Of course, no childhood in the early 2010s was complete without video games. While I loved playing them, my curiosity soon led me down a different path. Browser games were so popular back then, and I discovered the magic of the "inspect element" tool. It sounds fancy, but all I knew was that it let me mess with the on-screen content. Suddenly, I could change the amount of in-game currency (though, sadly, it wouldn't save!). It was a small act, but it marked my first step into the world of programming. It wasn't until my early high school years that I truly dove into coding. A friend introduced me to HTML, and I immediately took to it. I spent hours at home experimenting with HTML, captivated by the process of bringing web pages to life.

Unsupported ❌

Despite my growing passion for technology and coding, there were challenges along the way. My family, like many others in the Philippines, held doubts about the job prospects in the field of information technology (IT) or computer science. There was a prevailing belief that graduates in these fields would struggle to find employment, leading to a fate often referred to as "tambay" or being jobless. Despite these doubts, I remained steadfast in my pursuit of a career in technology, driven by my belief in its potential to change lives and make a positive impact on the world.

Today🌻

Fast forward to today. Despite being a late bloomer, I've achieved more than I ever thought possible. Was able to graduate with latin honors, I have also received a slew of awards for my coding skills and research, and was able to connect with different people all over the world. I am a living testament that can prove that hard work pays off.

As I look to the future, I have some questions for my future self. Which path did you choose – data engineering or full-stack engineering? And what are you doing now? I trust that you've made the right decision and are thriving in your chosen field.

In the meantime, I'll continue to give it my all, knowing that the best is yet to come. From toy blocks to code blocks, my journey into tech has been nothing short of remarkable, and I can't wait to see where it takes me next.

Aivan Carlos Tuquero⭐

Typescript Modules 101 A Minified Crash Course

Aivan Carlos Tuquero — Wed, 27 Mar 2024 14:58:04 +0000

TypeScript, the superset of JavaScript, brings much-needed structure and type safety to web development. A core concept in achieving this is the concept of modules. In this article, we'll delve into the theory behind modules in TypeScript, equipping you to write clean, maintainable, and scalable code.

What are Modules?

Imagine a large house. To navigate it efficiently, you wouldn't want to find everything in one giant room. Instead, you'd have separate rooms (modules) for specific purposes: a kitchen for cooking, a bedroom for sleeping, and so on. Modules in TypeScript work similarly. They encapsulate related code (functions, variables, classes) into self-contained units, promoting code organization and reusability.

Benefits of Using Modules:

Organization: Modules prevent code from becoming a tangled mess. They keep related functionality grouped, making the codebase easier to understand and navigate.
Reusability: Modules can be imported and used in other parts of your application. This reduces code duplication and promotes a modular development style.
Encapsulation: Modules can control access to their internal components using access modifiers (public, private). This helps prevent accidental modification and promotes data integrity.

Types of Modules:

TypeScript offers two main types of modules:

Namespace Modules: These modules declare a global namespace that groups related identifiers. They are useful for organizing large codebases but can lead to naming conflicts in complex projects.

namespace Math {
  export function add(x: number, y: number): number {
    return x + y;
  }

  export function subtract(x: number, y: number): number {
    return x - y;
  }
}

Module Scripts: These are the more common approach in modern development. They use the export and import keywords to define and access functionalities within a single file.

// math.ts
export function add(x: number, y: number): number {
  return x + y;
}

export function subtract(x: number, y: number): number {
  return x - y;
}

// main.ts
import { add, subtract } from './math';

console.log(add(5, 3)); // Output: 8
console.log(subtract(10, 2)); // Output: 8

Advanced Concepts:

Default Exports: Modules can have a single default export, typically a function or class.

// math.ts
export default function add(x: number, y: number): number {
  return x + y;
}

// main.ts
import add from './math';

console.log(add(5, 3)); // Output: 8

Import Types: You can explicitly define the types of imported variables for better type safety.

// math.ts
export interface Point {
  x: number;
  y: number;
}

export function distance(p1: Point, p2: Point): number {
  // ...
}

// main.ts
import { Point, distance } from './math';

const p1: Point = { x: 1, y: 2 };
const p2: Point = { x: 3, y: 4 };

const distanceResult = distance(p1, p2);

By understanding and applying module concepts effectively, you can write robust, maintainable, and scalable TypeScript applications.

-Aivan Carlos Tuquero

Image by freepik

Beyond the Basics: Mastering JavaScript Closures for Powerful Code

Aivan Carlos Tuquero — Tue, 26 Mar 2024 13:16:38 +0000

JavaScript is often lauded for its accessibility, making it a great entry point for new programmers. But beneath the seemingly simple syntax lies a treasure trove of powerful concepts. Closures are one such concept that can elevate your JavaScript skills from basic functionality to elegant problem-solving.

In this article, we'll delve into the world of JavaScript closures, exploring what they are, how they work, and most importantly, how to leverage them to write cleaner, more effective code.

Demystifying Closures: Functions and Scope

At its core, a JavaScript closure is a function that remembers and has access to variables from its outer (enclosing) function's scope, even after the outer function has returned.

Imagine a function acting like a room. Inside this room (the function's scope), there might be variables holding valuable tools (data). Now, a smaller function is created inside this room (the closure). Even after the room is locked (the outer function returns), the inner function can still access the tools left behind!

The Power of Remembering: Benefits of Closures

Closures offer several advantages in JavaScript programming:

Data Privacy: By creating private variables within closures, you can encapsulate data, preventing unintended modification from outside code.
State Management: Closures excel at managing the state of an application. They can remember values across function calls, enabling features like dynamic content updates or user interactions.
Modular Code: Closures promote modularity by creating self-contained functions with access to specific data, improving code organization and reusability.

Unveiling the Magic: Creating Closures in JavaScript

Here's an example to illustrate how closures work:

function createGreeter(greeting) {
  // Greeting variable is stored in the outer function's scope
  return function(name) {
    // Inner function can access the greeting variable from the outer scope
    return `${greeting}, ${name}!`;
  }
}

const morningGreeting = createGreeter("Good Morning");
const eveningGreeting = createGreeter("Good Evening");

console.log(morningGreeting("Rizal")); // Output: Good Morning, Rizal!
console.log(eveningGreeting("Efren"));   // Output: Good Evening, Efren!

In this example, the createGreeter function returns a new function that remembers the greeting variable even after createGreeter has finished executing. This allows us to create greetings with different salutations.

Level Up Your JavaScript: Practical Applications

Now that you understand closures, here are some practical ways to use them:
Simulating Private Variables: Create a module pattern using closures to mimic private variables in languages that lack them by default.
Building Interactive Components: Utilize closures to manage the state of interactive elements on a web page, like remembering user selections or form data.
Implementing Modules: Break down complex code into smaller, reusable modules using closures to encapsulate functionality and data.

Embrace the Potential: Conclusion

JavaScript closures might seem complex at first, but with practice, they become a valuable tool in your programming arsenal. By understanding how closures work and their practical applications, you can write cleaner, more modular, and powerful JavaScript code.

This post has just scratched the surface of closures. Feel free to explore further and experiment with creating your own closure-based solutions! Happy coding!

-Aivan Carlos Tuquero