For years, most Large Language Models (LLMs) have started from the same assumption: Language is first broken into tokens, and computation happens on those tokens.
Whether it's BPE, SentencePiece, or WordPiece, the tokenizer remains one of the most fundamental components of the entire architecture.
But while working on a new experimental framework called NINMENI, we decided to ask a different question: What if tokenization isn't the only possible foundation?
🚀 A Different Starting Point
Instead of treating tokens as the primitive unit, NINMENI explores a different assumption:
1 Character = 1 ID
Not because character-level models are new. They are not. The interesting part is what happens after that assumption is made.
Rather than using characters merely as smaller tokens, the framework treats the character space itself as a native substrate. Meaning formation is considered a higher-level process that emerges from interactions inside the runtime, instead of being delegated to a tokenizer.
This changes where complexity lives:
- ❌ Traditional Architecture: Invests complexity into segmentation algorithms.
- ✨ NINMENI: Complexity shifts entirely into semantic organization.
📊 Why This Matters
Traditional pipelines often look like this:
Text ──> Tokenizer ──> Token IDs ──> Model
The experiment we're building explores something closer to:
Language ──> Character Space ──> Native Character Registry ──> Meaning Units ──> Model
Notice that the registry isn't just "a list of Unicode characters." It's intended to become a deterministic representation of the language space itself. That distinction became surprisingly important.
💡 An Unexpected Engineering Lesson
During implementation, we discovered something interesting.
Initially, we manually maintained a registry containing thousands of supported characters. Whenever a missing character appeared, our instinct was: "Just patch the registry."
That turned out to be the wrong abstraction. The better approach was to generate the registry from a specification instead of maintaining it by hand.
The pipeline became:
Language Specification
↓
Writing Systems
↓
Relationships
↓
Unicode Coverage
↓
Registry Generator
↓
Native Registry
Now, the registry is:
- 🛠️ No longer handwritten.
- 🔄 Fully reproducible.
- 🔢 Deterministic.
- 🔍 Auditable.
Adding a new language doesn't mean editing thousands of entries manually anymore. It simply means extending the specification and regenerating the registry.
🛠️ Production Before Theory
One critical design decision we made was separating three distinct concerns:
- Paradigma: The conceptual rules.
- Mechanism: The operators that implement those rules.
- Runtime: The production infrastructure that executes them.
Keeping these layers completely independent allows the runtime to evolve without rewriting the conceptual model every time implementation details change.
💭 Why We're Sharing This
This isn't a claim that tokenization is obsolete. Nor is it a claim that we've discovered a universally better architecture.
It's an engineering experiment driven by one core question: Can a language model be built from a different primitive while remaining practical enough for production?
Whether the answer is ultimately "yes" or "no," exploring alternative assumptions is valuable. Many breakthroughs in computing started with someone questioning a design decision that everyone else had quietly accepted.
Perhaps tokenization is one of those decisions. Perhaps it isn't.
Either way, the only reliable way to find out is to build the system, measure it honestly, and publish what works and what doesn't.
👇 What Do You Think?
I'd be highly interested in hearing how other engineers would approach this problem.
If you were designing an LLM from scratch today, which architectural assumption would you challenge first? Let's discuss in the comments below!
Top comments (0)