DEV Community

kohei matsunobu
kohei matsunobu

Posted on Originally published at zenn.dev

I Built a Tiny Japanese GPT from Scratch in PyTorch and Tried a Toy 'RSI' Loop: Here's What It Does and Doesn't Show

Introduction

I'm a second-year student at ZEN University studying intelligent informatics (online), and I build AI agents as a personal project. I have no work experience in machine learning.

I use LLMs through APIs and agents every day, but I wanted to see what happens inside one with my own hands. So I wrote a small Japanese GPT using only basic PyTorch operations: no pretrained weights, no high-level model libraries. Then I tried a toy version of "RSI" (recursive self-improvement): the model generates several answers, grades them itself, and retrains on the best ones.

Up front: this is a tiny educational model trained on a few KB of text. Don't expect it to be useful, and please don't read the RSI numbers as "the model got smarter." I'll explain why below. This is not AGI or anything close to it.

What I built

A character-level, decoder-only Transformer with about 0.9M parameters (the pretraining version has 898,425, which I also checked by hand from the layer sizes). It has three stages:

  1. mini_gpt.py: character tokenizer + Transformer, pretrained on short excerpts of modern Japanese literature to continue text.
  2. chat_gpt.py: the same architecture, fine-tuned on ユーザー: ...\nAI: ...◁ dialogues so it behaves like a chatbot.
  3. rsi_gpt.py: a best-of-N self-training loop.

There is also a local web UI (web_ui.py, standard-library http.server only) that shows the top-5 probabilities for the first generated character and lets you edit the dialogue data and retrain in the browser.

How it works

1. A minimal Transformer

Hyperparameters in mini_gpt.py:

BLOCK_SIZE = 64       # max context length
N_EMBD = 128          # embedding dimension
N_HEAD = 4            # attention heads
N_LAYER = 4           # Transformer blocks
Enter fullscreen mode Exit fullscreen mode

Causal self-attention masks out future characters with a lower-triangular mask before the softmax:

att = (q @ k.transpose(-2, -1)) * (1.0 / math.sqrt(head_dim))
att = att.masked_fill(self.mask[:, :, :T, :T] == 0, float("-inf"))
att = F.softmax(att, dim=-1)
Enter fullscreen mode Exit fullscreen mode

Each block is Pre-LayerNorm with two residual connections:

def forward(self, x):
    x = x + self.attn(self.ln1(x))  # residual + attention
    x = x + self.mlp(self.ln2(x))   # residual + feed-forward
    return x
Enter fullscreen mode Exit fullscreen mode

The tokenizer just numbers the characters that appear in the training text:

chars = sorted(list(set(text)))
stoi = {ch: i for i, ch in enumerate(chars)}
Enter fullscreen mode Exit fullscreen mode

2. Turning it into a chatbot

I didn't change the architecture. Two things changed:

  • The input is formatted as "ユーザー: {question}\nAI: ".
  • Every AI answer in the training data ends with a stop character ◁, and generation stops when it is sampled.
if token_id == stop_token_id:
    break  # stop when the end marker appears
Enter fullscreen mode Exit fullscreen mode

Without the stop character the model keeps writing the next ユーザー: ... turn by itself. I call that role leakage below.

3. The RSI loop

This is a small-scale best-of-N rejection sampling loop, in the spirit of STaR / ReST:

  1. For each of 7 prompts, generate N=6 candidate answers.
  2. Score each one out of 100 with a rule-based function.
  3. If the best candidate scores 75 or more, keep that (prompt, answer) pair as synthetic data.
  4. Add the synthetic data to the dialogue data and retrain for 350 steps. Repeat for 3 generations.

The score is the sum of four parts (evaluate_candidate):

Part Points Meaning
Completion 25 ended naturally with ◁
Format 25 no "ユーザー"/"AI:" leakage, no newline, no 4+ repeated characters, at least 12 characters
Relevance up to 30 contains keywords I hand-picked per prompt (15 points each)
Self-confidence up to 20 the model's own average probability of the characters it sampled, times 20
matched_kw = [kw for kw in prompt_info["keywords"] if kw in reply]
kw_score = min(30.0, len(matched_kw) * 15.0)
...
conf_score = round(candidate["avg_confidence"] * 20.0, 1)
Enter fullscreen mode Exit fullscreen mode

The exploration temperature is lowered each generation: max(0.45, 0.85 - (gen - 1) * 0.12), i.e. 0.85, 0.73, 0.61.

Results

Sources: PAPER.md and rsi_history.json. I recomputed the RSI table from rsi_history.json.

Pretraining (1,678 characters of Japanese literature excerpts, 377 distinct characters, 800 steps): loss went from 6.1098 at step 1 to 0.0683 at step 800. Prompted with "吾輩は", it produces the opening of I Am a Cat.

Dialogue fine-tuning (27 Q&A pairs, 1,575 characters, 1,000 steps): loss went from 5.5833 at step 1 to 0.0387 at step 1000.

RSI (7 prompts, N=6, threshold 75, 350 steps per generation):

Generation Temperature Avg best score Avg worst score Prompts with worst < 75 Loss after retraining
Gen 1 0.85 97.56 80.77 4/7 0.0635
Gen 2 0.73 97.84 97.80 0/7 0.0490
Gen 3 0.61 97.84 97.80 0/7 0.0639

Gen 1 failures looked like this:

  • For "こんにちは!元気?" ("Hello! How are you?"), one candidate answered "こんばんは!…" ("Good evening!…") (69.8).
  • For "LLMとAttentionについて教えて", one candidate wrote "Attentionって何?\nAI: …" inside its own answer, i.e. role leakage (74.9).

The average worst-case score went from 80.77 to 97.80, and prompts with a failing candidate went from 4/7 to 0/7. But please don't read this as the model getting smarter. Here is why.

Limitations (please read this part)

  • Very little data. Pretraining text is 1,678 characters (about 5 KB); the dialogue set is 27 pairs. Losses of 0.04 to 0.07 mostly mean the model has memorized its data. I never held out a validation set, so there is no generalization number anywhere in this project.
  • The evaluation prompts are the training prompts. The 7 accepted synthetic pairs are answers to exactly the 7 prompts used for scoring (they sit at the end of chat_data.txt). Gen 2 and Gen 3 are being asked questions whose answers they saw in training. Higher scores are expected and say nothing about unseen questions.
  • The grader is the model itself plus my rules. The score comes from rules I wrote, keywords I chose, and the model's own confidence. Self-grading is biased toward whatever fits the rules, and the confidence term tends to reward memorized answers.
  • A confound. I also lowered the sampling temperature each generation, which by itself should reduce garbage output. I did not separate the effect of RSI from the effect of temperature.
  • One recorded run. I don't have repeated runs with different seeds, or a comparison against the same setup without RSI.
  • Not useful. Ask about anything outside the training data and you get an off-topic reply. The README says the same.

So what actually happened is: a tiny model picked its best-scoring answers to 7 questions and re-learned them.

Running it

pip install -r requirements.txt   # torch>=2.0.0

python mini_gpt.py                # pretrain and generate
python chat_gpt.py                # train on dialogues (saves chat_model.pt)
python chat_gpt.py --chat         # chat with the saved model
python web_ui.py                  # local web UI (http://localhost:8765)
python rsi_gpt.py                 # run 3 RSI generations
Enter fullscreen mode Exit fullscreen mode

Note that rsi_gpt.py rewrites chat_data.txt, so back it up first. If you only want to poke at it, the demo Space below is easier.

Takeaways

  • A ~0.9M-parameter Japanese GPT fits in plain PyTorch; mini_gpt.py is 175 lines including comments.
  • A prompt format plus a stop character was enough to turn a text-continuation model into a (very small) chat model, with no architecture change.
  • The RSI numbers come from a toy setup with self-grading and evaluation prompts leaked into training, so they are not evidence of general self-improvement. If I do this again, I would split off held-out prompts, use a grader other than my own rules, and fix the temperature.

Links

Top comments (0)