DEV Community: nlp

Day 36: Fine-Tuning BERT for Downstream Tasks

Priyesh Dave — Wed, 14 Oct 2026 13:55:38 +0000

Previously, on Day 35: Explained how BERT uses bidirectional transformer encoders and masked language modeling to create contextualized word representations, enabling state-of-the-art understanding and prediction in NLP tasks. Introduced the concepts of tokenization, attention, fine-tuning, and provided a code example for masked token prediction.

Why Fine-Tune BERT?

Fine-tuning BERT means adapting a large, pretrained language model to solve a specific problem. For example, you might want BERT to tell if a movie review is positive or negative, extract place names from news articles, or answer questions about a paragraph.

This is not the same as training a language model from the ground up. Training from scratch means starting with random weights and teaching a model how language works. Fine-tuning, by contrast, starts with a model that already understands general language patterns and vocabulary. You only nudge its knowledge toward your specific task.

Pretraining a model like BERT is expensive—it requires vast text datasets and powerful hardware for days or weeks. Fine-tuning is much lighter. You only need a task-specific dataset, which is usually much smaller, and the process takes hours or less. The pretrained BERT already "knows language"; you just show it how to use that knowledge for your problem.

Understanding Downstream Tasks

A downstream task is any real-world language problem you want BERT to solve. Think of these tasks as the jobs you give BERT once it finishes general pretraining.

Some concrete examples:

Text Classification: Decide if a message is spam or not, or if a review is positive or negative.
Sentence Pair Classification: Given two sentences, predict if one entails the other (natural language inference), or if they mean the same thing (paraphrase detection).
Named Entity Recognition (NER): Find and categorize mentions of people, places, or organizations in text.

Pretraining focuses on general language—predicting missing words or the next sentence. Downstream tasks use labeled data—examples where you know the right answer—to teach BERT a new skill.

How Fine-Tuning Works: High-Level Steps

Fine-tuning BERT for a downstream task follows a clear recipe:

Load a pretrained BERT model.
Add a classification "head": Attach a small, task-specific output layer. For binary tasks, this is often just one linear layer.
Gather task-specific data: Use labeled examples pairing text with the correct answer.
Train with supervision: Run a standard training loop. Compute loss comparing model outputs to the true labels. Update the model’s parameters—including BERT’s existing weights—to reduce that loss.

During this process, BERT adapts its language knowledge to fit your task. The only new parameters are the small classification head. Most of the model remains unchanged, but its deep understanding of language is steered toward your problem.

Fine-Tuning BERT for Text Classification: A Minimal Example

We’ll fine-tune BERT to classify movie reviews as positive or negative. We'll use the Hugging Face Transformers library, which simplifies the process.

First, install the required libraries:

pip install torch transformers datasets

Example Code

from transformers import BertForSequenceClassification, BertTokenizer, Trainer, TrainingArguments
from datasets import load_dataset

model_name = "bert-base-uncased"
model = BertForSequenceClassification.from_pretrained(model_name, num_labels=2)
tokenizer = BertTokenizer.from_pretrained(model_name)

# Load a small sample of the IMDb movie reviews dataset
dataset = load_dataset("imdb", split="train[:2%]")  # Small subset for speed

def tokenize(example):
    return tokenizer(example["text"], padding="max_length", truncation=True, max_length=128)

tokenized_dataset = dataset.map(tokenize, batched=True)
tokenized_dataset.set_format(type="torch", columns=["input_ids", "attention_mask", "label"])

training_args = TrainingArguments(
    output_dir="./results",
    num_train_epochs=1,
    per_device_train_batch_size=8,
    evaluation_strategy="epoch",
    logging_steps=10,
    save_steps=10,
    logging_dir="./logs",
    no_cuda=True  # Remove if you have a GPU
)

trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=tokenized_dataset,
    eval_dataset=tokenized_dataset[:100]  # Use a small slice for evaluation
)

trainer.train()

result = trainer.evaluate()
print(f"Eval loss: {result['eval_loss']:.3f}, Accuracy: {result['eval_accuracy']:.3f}")

Step-by-step:

Load pretrained BERT and its matching tokenizer.
Fetch a small subset of IMDb reviews, each labeled as positive (1) or negative (0).
Tokenize the reviews. This converts raw text into input IDs BERT can process.
Use Hugging Face’s Trainer to handle batching, loss calculation, and parameter updates.
Fine-tuning adjusts both BERT and the attached head using your labeled examples.
Evaluate on a small held-out set to check results.

For real-world use, split data into proper train and validation sets, and train longer.

Avoiding Common Fine-Tuning Pitfalls

Fine-tuned models often show big improvements on your labeled data, but you need to watch out for problems common in transfer learning:

Overfitting: The model does well on training data, but poorly on new data. Fix by using more data, applying regularization, or stopping training earlier.
Learning Rate Problems: With BERT, training often works best with a low learning rate—commonly 2e-5 to 5e-5. Too high, and the model unlearns its language knowledge. Too low, and updates are slow.
Bad Labels: If your labeled data is messy or incorrect, your model’s predictions will be too.
Tokenization Issues: Make sure text is preprocessed to match what BERT expects. Truncate or pad consistently.
Resource Limits: Fine-tuning uses memory. If you run out of RAM or GPU memory, lower your batch size or sequence length.

Best practices:

Track both training and validation loss/accuracy.
Always evaluate on a separate validation set, not just training data.
Save model checkpoints throughout training.

Fine-tuning BERT unlocks strong performance on a wide range of NLP tasks. With just a bit of labeled data and clear attention to the process, you can adapt state-of-the-art language models for your own problems.

Key Takeaways

Fine-tuning adapts pretrained BERT for specific tasks using small, labeled datasets.
Pretraining BERT is resource-intensive, but fine-tuning is lightweight and task-focused.
Common downstream tasks include text classification, sentence pair classification, and named entity recognition.
A classification head is added on top of BERT for prediction during fine-tuning.
Careful data handling, low learning rates, and validation are crucial to avoid overfitting or training issues.

Try It Yourself

Download a small labeled sentiment dataset (like a CSV with a handful of sentences and their labels). Fine-tune a BERT model for sentiment classification using the Hugging Face Transformers example code. After training, test the model on a few custom sentences of your own to observe how its predictions change after fine-tuning.

Further Resources

Coming up on Day 37: GPT Architecture: Decoder-Only Transformers

I Thought Extracting Dates Was Easy Until I Built an On-Device iOS App

osman reşat doğrul — Wed, 14 Oct 2026 08:56:15 +0000

I started building an iPhone app to help identify tasks and follow-ups hidden inside everyday messages.

Appointments, payments, delivery notices, things someone asks you to do by Friday. The kind of messages you read, understand, and then sometimes forget about.

At first, I thought extracting dates and finding actionable text would be the hardest part.

I was wrong.

Finding a date isn't the same as understanding it

Consider this message:

"Your appointment is on October 15. Please confirm by October 12."

Two dates, but two different purposes.

October 15 describes an event. October 12 is the deadline for an action.

A date extractor can identify both correctly and still fail to understand what matters.

In NLP terms, extracting dates is only the first step. The harder part is identifying the intended action and understanding how each date relates to it.

A useful interpretation might look like this:

{
  "event": "Appointment",
  "event_date": "2026-10-15",
  "action": "Confirm attendance",
  "action_deadline": "2026-10-12",
  "action_required": true
}

Conceptual example only — not an actual Reply Radar model response or internal data schema.

The difficult part isn't producing structured data. It's making sure the structure reflects what the message actually means.

And then there's another problem: not every message actually requires the recipient to do something.

A delivery confirmation, a completed payment notification, and a request for payment might contain similar words. But they shouldn't all become tasks.

That's where I started spending more time than I expected: distinguishing useful information from something that actually needs attention.

Starting with Turkish

I began with Turkish, my native language, and most of my testing has been in Turkish.

One interesting part of Turkish is how much meaning can fit into a single word.

Take "konuşuruz," roughly meaning "we'll talk."

Depending on the context, it might suggest a future conversation. Or it might just be a polite way to end one.

Would you create a reminder from that?

Not necessarily.

The same problem exists in English, but the expressions and grammatical structures differ.

Reply Radar can analyze English messages too. The challenge isn't simply adding English support. It's making the interpretation reliable across languages and different ways people express intent.

Why I chose on-device processing

Messages often contain information people would rather keep private.

That's why I wanted users to decide which message text or screenshot to share with the app.

The analysis runs on the iPhone. Raw message content isn't sent to a server for analysis, and the app doesn't monitor conversations in the background.

I'm relying heavily on Apple's built-in on-device capabilities.

But keeping processing on the device doesn't automatically make language understanding easy.

Recognizing the words is one thing.

Understanding whether those words represent an unfinished task is another.

Thinking about Apple's evolving AI models

Apple's on-device AI developments, including the evolution of Foundation Models in iOS 27, are interesting for developers working on problems like this.

But I see a few challenges worth paying attention to.

What happens when a model interprets a deadline as an event date?

How should an app respond when the context is ambiguous?

What happens when a language has different levels of model support?

And how do we keep the results predictable as the underlying models change?

These are questions I'm continuing to explore, not problems I consider fully solved.

In my experience, getting a model to return a plausible answer is only the beginning. Deciding whether that answer represents the user's actual intention is much harder.

A small problem, but an interesting one

The app I've been building is called Reply Radar.

It's not intended to replace a calendar or a full task manager.

The goal is narrower: help people notice the tasks, dates, and follow-ups that might otherwise remain buried in messages.

I'm still learning where this approach works well and where it needs improvement.

What surprised me most was how quickly a fairly small productivity idea turned into an exercise in language, context, and reliable on-device analysis.

And that's probably what has made the project worth continuing.

For anyone curious about the project, I've put together a small website explaining what Reply Radar does: Reply Radar — Project Website.

Writing note: This article was prepared with AI writing assistance and is based on my experience developing and testing Reply Radar.

What Changes When AI Text Is Humanized: 138 Pairs, Measured

Emir Vatric — Wed, 14 Oct 2026 05:34:06 +0000

Originally published on the ToHuman blog: https://tohuman.io/blog/what-changes-when-ai-text-is-humanized-2026 — that page carries the dataset JSON-LD and is the canonical version.

Read this first, because it changes what the numbers mean. The texts below are
not our customers' texts. They are a corpus we wrote ourselves, put through the same
production endpoint a paying user hits. We did it that way because our own Terms do
not permit the other version — the detail is in the method, and it is the second
finding of this piece, not a disclaimer at the bottom.

The one question nobody has measured

Every study of machine-written prose compares two categories. Human text on one
side, AI text on the other — em-dash counts, burstiness, perplexity, the vocabulary
that models over-reach for. There are dozens of these. They are the basis of every
detector pitch and every "how to spot AI writing" listicle.

There is a third category and almost no public measurement of it: the humanized
pair — one text, before and after a rewrite that was asked to make it read like a
person wrote it. Not AI vs human. AI vs the-same-AI-text-rewritten. Nobody outside
the small set of companies that run these rewrites holds that pair, and as far as we
can find, none of them has published the distributions.

So we measured ours. 138 English before/after pairs across two corpora, scored with
the same word lists and thresholds that power our public
AI Tells Scanner — no new regexes invented for the occasion.

Three things are worth knowing before the tables:

The median AI-tell density goes from roughly 12–14 events per 1,000 words to 0.
For about a third of the texts there was nothing there to move in the first place.
Sentence rhythm gets burstier in roughly two thirds of texts and flatter in roughly three in ten — a failure direction large enough that an average would hide it.

No AI detector was run. Not one. What that excludes is spelled out below.

Method

N. 138 English pairs, in two corpora scored separately because they disagree on
magnitude and we would rather publish the disagreement than average it away.

	Corpus A	Corpus B
Rewritten	2026-09-14	2026-09-24
Records in file	76	80
Excluded — not a completed rewrite	0	8 (rejected by our own quality rails)
Exact-duplicate inputs removed	0	0
Pairs	76	72
English pairs (headline set)	66	72
Language split	en 66 · es 5 · fr 5	en 72
Median input length (words, all pairs)	77.5	49.5
Length range (words, all pairs)	18 – 861	26 – 247

Window. Both corpora were rewritten in September 2026, ten days apart, by the
production endpoint at intensity=medium, multi-pass. The model and pipeline config
may differ between the two runs and from today's production. That is a real
limitation, not a hedge.

Exclusions, all of them. Eight corpus-B records never completed a rewrite — they
were refused by our own quality rails, so they are out of the headline. (A 10%
rejection rate on a corpus we wrote is itself a product signal; we are not filing it
as data cleaning.) Four other bench runs were rejected outright because output == input on every row — input-echo artefacts, not rewrites. Two further runs had real
pairs but ran on a local build rather than production, so they are out of scope. The
ten Spanish and French pairs are excluded from every headline, because every slop and
cliché list in the scanner is English: they score 0.0 both before and after, which
means "not measured," not "clean."

Aggregate only. No text is reproduced anywhere in this piece or in the published
raw data. No per-item or per-account figures. Every number is a distribution — median,
interquartile range, and the share of pairs moving each direction.

Why the corpus is synthetic, verbatim. Before computing a single statistic we read
our own live policies. The decisive line, from Terms §6:

"We process your text solely to provide the humanization service and do not use
your content to train our models."

solely is an exhaustive-purpose limitation, and computing style statistics for a
public report is not providing the service to the person who submitted the text. The
only research-shaped permission in the privacy policy is for "aggregate, anonymized
usage statistics (never individual content)" — and our method requires reading
individual content to produce each per-pair measurement, even though only aggregates
get published. So the verdict was recorded as NOT PERMITTED → synthetic corpus
used, and from the source report:

"This is not first-party user data. It is our own synthetic corpus run through the
production endpoint, used because our own Terms and Privacy Policy do not permit the
first-party version. […] Real user text differs in genre mix, length, language, prompt
style and messiness. Treat every figure as 'what the production model does to text
we wrote,' not 'what it does to what customers submit.'"

No production query against the stored input or output columns was run at any point.
The thing this substitution costs us is specific and worth naming: there is no
api / playground / UI split in what follows, because that dimension only exists in the
table we did not query.

What changed

Rates are per 1,000 words. down / flat / up is the share of pairs moving that
direction, with a small dead band (0.1 per 1,000 words; 0.01 for the CV) so that
4.01 → 4.00 counts as flat rather than as an improvement.

Composite AI-tell density — the one that moves

The composite is the plain sum of four scanner rate checks: slop vocabulary, cliché
phrases, em dashes, three-item lists. It is our own construction, unweighted and
untrained — a different weighting would move the failure shares later in this piece.

	median in	IQR in	median out	IQR out	Δ median	down	flat	up
Corpus A (n=66)	12.26	0 – 16.05	0	0 – 6.51	−4.51	68.2%	30.3%	1.5%
Corpus B (n=72)	13.68	0 – 44.40	0	0 – 17.63	−2.09	51.4%	40.3%	8.3%

Read the IQRs, not just the medians. The lower quartile is 0 in both corpora going in —
a median that moves 12.26 → 0 is partly a statement about how concentrated the tells
were, and not only about how hard the rewrite worked.

Cliché phrases — the single most-moved measure in the study

	median in	IQR in	median out	IQR out	Δ median	down	flat	up
Corpus A (n=66)	2.15	0 – 11.88	0	0 – 0	−1.60	59.1%	40.9%	0.0%
Corpus B (n=72)	0	0 – 1.15	0	0 – 0	0	22.2%	73.6%	4.2%

59.1% of corpus-A pairs shed cliché density and none gained any. This is the
cleanest directional result anywhere in the data, and it is also the narrowest: corpus
B's inputs barely contained clichés, so there was nothing to remove.

Slop vocabulary

	median in	IQR in	median out	IQR out	down	flat	up
Corpus A (n=66)	0	0 – 0	0	0 – 0	18.2%	78.8%	3.0%
Corpus B (n=72)	0	0 – 30.80	0	0 – 0	33.3%	61.1%	5.6%

Corpus B's upper quartile falls from 30.80 to 0 — the slop reduction is real, but it
lives in the upper tail of texts that had slop to lose: 33.3% of pairs moved down and
61.1% had nothing to move.

Sentence rhythm (coefficient of variation) — up is the intended direction

	median in	IQR in	median out	IQR out	Δ median	up	flat	down
Corpus A (n=39)	0.40	0.29 – 0.52	0.47	0.36 – 0.55	+0.04	69.2%	5.1%	25.6%
Corpus B (n=30)	0.33	0.24 – 0.43	0.41	0.31 – 0.47	+0.03	60.0%	10.0%	30.0%

Note the n. This is measured on 39 of 66 and 30 of 72 pairs, because the scanner
refuses to judge rhythm below five sentences. The CV figures describe the longer half
of the corpus and nothing else. The down column gets its own section below.

Length — humanized text comes back slightly longer

	median word count in	median out	median length Δ	IQR of Δ	longer	shorter
Corpus A (n=66)	74	78	+3.23%	−3.49% – +10.02%	62.1%	33.3%
Corpus B (n=72)	49.5	55	+5.03%	−2.00% – +14.27%	68.1%	30.6%

If you are budgeting word count, API spend, or a fixed-length slot, plan for the
rewrite to add a few percent. Mean sentence length barely moves either way (A: 17.05 →
17.80 words; B: 15.50 → 16.25), with pairs split roughly in half on direction.

Paragraph structure — never touched, 138 of 138

	pairs	paragraph count changed
Corpus A	66	0
Corpus B	72	0

A clean negative result and the most practically useful one in the set: the rewrite
works inside the shape you give it. Opener repetition, measurable on the 18 corpus-A
pairs with three or more paragraphs, also did not move (2 in, 2 out); corpus B has no
multi-paragraph items, so it is unmeasurable there.

What barely changed, or got worse

Same prominence as the section above, because these are results.

	Corpus A	Corpus B
English pairs scored	66	72
Inputs with a composite of 0 — nothing for these checks to fix	20 (30.3%)	28 (38.9%)
Pairs where a % change in the composite is even definable	46	44
Barely moved (\|Δ\| / composite in < 5%)	2 of 46 — 4.3%	5 of 44 — 11.4%
Got worse (composite rose)	1 of 66 — 1.5%	7 of 72 — 9.7%
Zero-tell inputs that acquired tells	1	2
Got less bursty (sentence rhythm flattened)	11 of 39 — 28.2%	9 of 30 — 30.0%

The honest headline is the range, not the friendlier end: the statistics got worse
for 1.5% to 9.7% of texts, and barely moved for another 4% to 11% — and for about
three in ten, the rewrite flattened the sentence rhythm it was supposed to vary.

Then there is the result that matters most and flatters us least.

	Corpus A	Corpus B
Texts the scanner flagged on ≥1 pattern, before	24	27
Texts the scanner still flags on ≥1 pattern, after	19	18
Cleared	5 of 24	9 of 27

On our own scanner's conservative thresholds, most text that read as machine-written
still reads that way afterwards. The density improves for the majority of pairs; the
verdict usually does not flip. Those two sentences are both true at the same time, and
any version of this report that published only the first one would be dishonest.

Two more measures we are reporting as untested rather than unchanged, because the
corpus did not contain enough of them to test:

Em dashes. 98.5% of corpus-A pairs and 90.3% of corpus-B pairs had no em-dash change, because there were almost no em dashes going in. We cannot claim the model removes them. The most-repeated tell in the entire genre, and we have nothing on it.
Contractions. 86.4% and 65.3% flat, same reason. We cannot claim the model adds them.

What this does and does not say

No AI detector was run. Not GPTZero, not Originality, not Turnitin, nothing. Every
number above is a style measurement taken with our own published word lists.

Style change is not a detection outcome. The two are not known to be the same axis
and we did not test the link. Nothing in this report supports a claim about detection,
"bypassing," or "undetectability" — if you arrived looking for evidence that rewriting
text defeats a detector, this study does not contain it, and we would rather say so
plainly than let the composite table imply it.

This is not user data. See the method. It is our own corpus through the production
endpoint, because our policies do not permit the first-party version. Real submissions
differ in genre, length, language, prompt style and messiness.

n is small — 66 and 72 English pairs — and the two corpora disagree on magnitude.
Where they disagree we have published both.

The composite is ours. Nobody else uses it. It is a plain sum of four published
per-1,000-word rates, and a different weighting would move the "barely moved" and "got
worse" shares. The inputs are listed in the appendix so you can re-weight them.

Non-English is not measured at all.

How to use it

If you write. Three findings are directly actionable. Clichés are what a rewrite
reliably removes, so they are also the thing worth removing yourself first. Paragraph
structure survives untouched, so the shape you hand over is the shape you get back.
And output runs 3–5% longer at the median — budget for it. ToHuman is the rewrite tool
these pairs came from, and the honest scope of what it rewrites is prose: body copy,
email, documentation, and for anything code-shaped, the comments, docstrings, READMEs,
commit messages and PR descriptions around the code — never the code itself. We
measured that boundary separately and wrote up where it holds and where it breaks in
can you humanize AI-generated code.

If you edit. The verdict table is the one to keep. A tool can cut measured tell
density by most of its range and still leave text that trips the same pattern checks —
5 of 24 and 9 of 27 cleared. Treat a rewrite as a first pass, not a finish line, and
re-read for the specific thing you were worried about. Our comparison of the tools in
this category, including what each one actually claims, is at
best AI humanizer tools 2026; the
AI Tells Scanner runs the same checks used here, in your browser,
on your own text.

If you research this. The pair corpus is the missing third category in the AI-vs-human
literature and we would like more of it to exist. The scanner source
(public/ai_tells_scanner.js) is public, so the measurement is reproducible against any
corpus of pairs: 90 slop patterns, 196 cliché phrases, 4 cliché regexes, plus em-dash,
sentence-uniformity, triad and opener-repetition checks with fixed thresholds. If you are
generating pairs at scale programmatically, the
humanizer API is the same endpoint these runs used. Two gaps are
cheap to close and we would welcome someone closing them first: a testset that actually
contains em dashes and contractions, and a detector arm on the same pairs.

Credit line. If you cite these numbers:

ToHuman, "What changes when AI text is humanized: 138 pairs, measured" (October 2026).
https://tohuman.io/blog/what-changes-when-ai-text-is-humanized-2026

Corrections and replications are welcome — including ones that contradict this.

Method appendix

Scoring code. Every AI-tell measure comes from the live AI Tells Scanner
(public/ai_tells_scanner.js), required unmodified into Node via its module.exports.
The published numbers therefore come from exactly the code that powers
/ai-tells-scanner: 90 slop-vocabulary patterns, 196 cliché phrases,
4 cliché regex patterns, and the scanner's own em-dash, sentence-uniformity, triad and
opener-repetition checks, with its own thresholds and its own sentence, paragraph and
word splitters. No word list or regex was re-invented for this report.

Composite. slop_1k + cliche_1k + emdash_1k + triads_1k — the plain sum of the
scanner's four rate-based checks in events per 1,000 words. Unweighted, untrained. The
scanner's other two checks are excluded deliberately: sentence-length CV is a ratio that
should move up, and opener repetition is a share of paragraphs. Both are reported
separately above.

Contraction rate. Locally defined: an allow-list of roughly 60 contracted forms
(don't, it's, we're, …) per 1,000 words. A generic \w+'\w+ pattern would score
every possessive as a contraction and would inflate formal academic prose most, so
possessive 's is excluded.

Direction shares. Dead band of 0.1 per 1,000 words for rates, 0.01 for the CV.
"Barely moved" = |Δcomposite| / composite_in < 5%, among pairs with composite_in > 0.
"Got worse" = composite_out > composite_in.

Burstiness n. The scanner will not judge rhythm under five sentences
(THRESHOLDS.minSentences), so the CV rows cover 39 of 66 (A) and 30 of 72 (B) pairs.

Genre mix. Corpus A: academic 34, blog 16, multilingual 10, personal 9, business 7;
tiered 56 paragraph-length / 20 document-length. Corpus B: long-control 20, blog-short 11,
email-short 11, business-note 8, essay-para 7, human-sourced controls 7, other 8.

Source. 100% bench-synthetic in both corpora, rewritten by the production RunPod
endpoint. Full statistics report and machine-readable output:
marketing/reports/humanization-corpus-stats-2026-10.md and its .raw.json.

Why Can't LLMs Count the R's in "Strawberry"? How Tokenizers Work

AI Engineering Illustrated — Tue, 13 Oct 2026 22:49:40 +0000

The preprocessing step behind spelling slips, shaky arithmetic, glitch tokens and your API bill

Ask a language model how many r's are in "strawberry" and, for a long time, the popular answer was "two." The same models could write a sonnet about strawberries, explain their genetics and translate the word into forty languages. So why does counting letters go wrong?

The model never sees the letters. Before your prompt reaches the first transformer layer, a separate program called the tokenizer chops the text into chunks and swaps each chunk for an integer. The model only ever reads those integers. If "strawberry" becomes a single integer, the three r's are hidden inside a number the model can't take apart.

Tokenization is the least glamorous part of the LLM stack, and it causes a surprising number of problems: shaky multi-digit arithmetic, higher costs for non-English users, strange "glitch" words that make models act up, and token counts that don't match your character-based estimates. In this issue we'll open the tokenizer up: how it turns text into IDs, how Byte Pair Encoding (BPE) learns its vocabulary, and how its choices show up as spelling, arithmetic, language and glitch-token problems you can design around.

Every token split in this issue was measured with OpenAI's open-source tiktoken library on the actual GPT-2, GPT-4 (cl100k_base) and GPT-4o (o200k_base) vocabularies, so you can reproduce them yourself.

Tokens: the Model's Only Alphabet

A neural network can't do math on strings. It needs numbers. The simplest scheme would give every character a number, but then a 2,000-word document becomes more than 10,000 positions, and every transformer layer pays for each one. The other extreme gives every word a number, but no fixed list covers every word, name, typo, URL and code identifier people will ever type.

Rico Sennrich and colleagues measured this trade-off in their 2016 paper Neural Machine Translation of Rare Words with Subword Units. On a German training corpus of about 100 million words, keeping whole words needed 1.75 million vocabulary entries and still left 1,079 unknown words in the test set. Splitting into characters needed only about 3,000 symbols but stretched the corpus to 550 million positions. Their subword method, BPE, landed in between: about 63,000 symbols, 112 million positions and zero unknown words.

That middle ground is a token: a chunk of text that's usually shorter than a word for rare words and exactly one word for common ones. Modern LLMs use vocabularies of 50,000 to 200,000 tokens. tiktoken's README puts the average at about 4 bytes of text per token. On the English text of last week's issue I measured 4.4 characters per token with the GPT-4o tokenizer, or roughly 0.63 words per token.

From Text to Token IDs in Five Steps

From raw text to token IDs: pre-split, apply BPE merges, look up IDs, then embed

Here's what happens to the sentence "I love strawberry" on its way into GPT-4o:

Raw text. Your string arrives as UTF-8 bytes. Some tokenizers normalize it first (Unicode normalization, lowercasing in older models), but GPT-style tokenizers keep the bytes exactly as typed.
Pre-split. A regular expression cuts the text into pieces that merges are never allowed to cross: letter runs, digit runs, punctuation, whitespace. The tiktoken source shows the patterns. GPT-2's version keeps a leading space attached to the following word, so the pieces are I, love, strawberry. That's why the dot in the figure matters: " strawberry" with a space and "strawberry" without one are different tokens.
BPE merges. Each piece starts as a sequence of bytes. The tokenizer then applies its learned merge rules, in the order they were learned, gluing adjacent pairs together until no rule applies. Common words collapse into one token. Rare ones end up as a few subword chunks.
Token IDs. Each final chunk is looked up in the vocabulary. With o200k_base, "I love strawberry" becomes [40, 3047, 101830]: three integers.
Embeddings. Inside the model, each ID selects one row of a learned embedding matrix, a vector of a few thousand numbers. From here on, the transformer only ever works with those vectors.

Two properties fall out of this design, and the tiktoken README calls both out. Tokenization is lossless: decoding the IDs gives back the exact original bytes. And it works on any text, even strings that never appeared when the tokenizer was trained. The second property depends on a trick we'll get to in a moment.

How BPE Learns Its Vocabulary

Byte Pair Encoding started life as a data compression algorithm (Philip Gage, 1994): repeatedly find the most common pair of adjacent bytes and replace it with a new, unused symbol. Sennrich et al. adapted the idea to build a vocabulary instead of compressing a file. Training goes like this:

Split every word in the training corpus into characters. The starting vocabulary is just those characters.
Count every pair of adjacent symbols, weighted by how often each word appears.
Merge the most frequent pair into a new symbol, add it to the vocabulary, and record the rule.
Repeat until the vocabulary reaches the target size.

The number of merges is the only real setting. As the paper puts it, the final vocabulary size equals the starting symbols plus the number of merge operations.

BPE training: count adjacent pairs, merge the most frequent, repeat until the vocabulary is full

The diagram uses the toy corpus from the Hugging Face LLM course: "hug" (10 times), "pug" (5), "pun" (12), "bun" (4) and "hugs" (5).

Start: the vocabulary is b g h n p s u.
Merge 1: the pair u + g appears 20 times (10 in hug, 5 in pug, 5 in hugs), more than any other, so ug becomes a token.
Merge 2: u + n now leads with 16 (12 in pun, 4 in bun), so un joins.
Merge 3: h + ug appears 15 times, and the first three-letter token, hug, is born.

Real tokenizers run this loop tens of thousands of times over gigabytes of text. GPT-2's vocabulary has 50,257 entries. GPT-4's cl100k_base has about 100,000 and GPT-4o's o200k_base about 200,000 (100,277 and 200,019 including special tokens, according to tiktoken). Meta says Llama 3 moved to a 128K-token vocabulary that yields up to 15% fewer tokens than Llama 2 on the same text.

Encoding new text replays the merges. Split the input into characters (or bytes), then apply the learned rules in priority order. A word seen often during training collapses into one token. A word never seen still gets encoded, just in smaller pieces. With GPT-4o's vocabulary, "unbelievably" becomes un | bel | ievably.

The order of merges also explains some quirks. Merges reflect the tokenizer's training data, which isn't necessarily the model's training data. Keep that in mind; it comes back when we get to glitch tokens.

Byte-Level BPE: No More "Unknown Word"

Classic BPE starts from characters, and Unicode defines roughly 150,000 of them. Any character missing from the training corpus becomes [UNK], the "unknown" token. The Hugging Face course notes this is one reason older NLP models handled emoji badly.

GPT-2 introduced a simple fix: run BPE over bytes instead of characters. There are only 256 possible byte values, so the base vocabulary has 256 entries and every possible input, whether emoji, Tamil script or binary garbage, can be written in them. Merges then build up common multi-byte sequences. As the course puts it, GPT-2 doesn't actually have an unknown token, because an unknown character can't occur with byte-level BPE.

You can see the byte fallback at work. In GPT-2's vocabulary, the emoji 🙂 (four UTF-8 bytes) becomes 2 tokens. GPT-4o's larger vocabulary learned the whole sequence, so it's 1 token. Either way, nothing is ever "unknown." The price is that a rarely seen script can get chopped into fragments that aren't even whole characters, which leads to the language tax below.

Why the Model Can't See the R's

Back to the strawberry. Here's what GPT-4o's tokenizer does with the classic question:

The model sees “ strawberry” as one ID with no letters inside; spelling it out or calling code exposes the letters

"How many r's are in strawberry?" becomes 8 tokens: How | ·many | ·r | 's | ·are | ·in | ·strawberry | ?. The word you care about is token #101830. To the model, that's one opaque symbol, mapped to one embedding vector. There's no "r" in it to count.

Whatever the model knows about how strawberry is spelled, it had to memorize from training text where the word happened to be spelled out, as in dictionaries, spelling-bee pages or kids' worksheets. That knowledge is thin and indirect, so the model guesses, and a plausible guess for a word ending in a double letter is "two."

The split also depends on context in ways people don't expect:

strawberry (leading space, the usual case mid-sentence) is 1 token.
strawberry (no leading space, for example at the start of a message) is 3 tokens: st | raw | berry.
Strawberry (capitalized) is a different 3: Str | aw | berry.

So the model may have learned slightly different things about the same word.

The fixes all expose the letters. Writing the word as s-t-r-a-w-b-e-r-r-y turns it into 10 tokens, one per letter, and now counting is something the model can actually do. Better still, when exact character-level answers matter (counting, reversing a string, checking a regex), have the model call a code tool. One line of Python, "strawberry".count("r"), returns 3 every time.

The Multilingual Token Tax

The tokenizer's merges come from its training data, and that data is mostly English and code. English words get dedicated tokens. Other languages, especially ones with their own scripts, get fewer merges and are left closer to raw bytes.

The token tax: the same sentence costs more tokens in some languages, and numbers split into left-aligned chunks

I ran one short sentence, "The weather is nice today," and simple translations of it through three of OpenAI's tokenizers. In English, it's 6 tokens in all three. In Tamil, it's 85 tokens with GPT-2's tokenizer, 42 with GPT-4's and 11 with GPT-4o's. Hindi drops from 27 to 21 to 5, and Russian from 27 to 13 to 6. That's just one sentence, not a benchmark, but the direction is clear. Larger, more multilingual vocabularies have closed much of the gap, though not all of it.

Aleksandar Petrov and colleagues studied this systematically in Language Model Tokenizers Introduce Unfairness Between Languages (2023). They found the same text can differ in token length by up to 15× across languages, and that even byte-level and character-level models show gaps of more than 4×. The consequences are concrete:

Cost: APIs bill per token.
Context: a 128K-token window holds far fewer pages of Tamil than of English.
Latency: output is generated one token at a time.

If you serve a non-English market, measure token counts on your actual traffic. Don't assume the "about 4 characters per token" rule of thumb.

Numbers: Why Long Addition Goes Wrong

The same pre-split regex decides how numbers are chunked, and labs have made very different choices. Aaditya Singh and DJ Strouse surveyed them in Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs (2024):

Pure BPE (GPT-3, Claude 2.1, OLMo): whichever digit strings happened to be frequent got tokens. 710 might be one token while 711 isn't, so numbers of the same length split inconsistently.
Single digits (PaLM, Llama 1 and 2, Mistral): every digit is its own token.
Chunks of up to 3 digits, left to right (GPT-3.5, GPT-4): the \p{N}{1,3} rule in the tiktoken pattern. GPT-4o's o200k_base uses the same rule.

Left-to-right chunking means 1234567 becomes 123 | 456 | 7. The chunks line up with the start of the number, while place value, and the way carries move when you add, runs from the end. Insert commas and the tokenizer splits on them, giving 1 | , | 234 | , | 567. Now every chunk sits on a thousands boundary.

Singh and Strouse tested exactly this on GPT-3.5 with 7- to 9-digit addition. With plain digits, 8-shot accuracy was 75.6%. With commas, it was 97.8%. When the answer had more digits than the inputs (because of a final carry), plain-digit accuracy fell to 8.25%, and the errors weren't random: the model almost always got the first three digits right and the fourth digit wrong, right at the first token boundary. Asking the model to first rewrite the problem with commas recovered most of the lost accuracy. GPT-4 was less affected, but the effect didn't disappear.

The lesson isn't "add commas everywhere." It's that number formatting changes the problem the model has to solve. For anything that must be exact, like invoices or financial math, route the arithmetic to a calculator or code tool.

Glitch Tokens: When the Vocabulary Outlives the Data

In early 2023, Jessica Rumbelow and Matthew Watkins went looking at GPT-2 and GPT-3's token embeddings and found a cluster of strange entries: SolidGoldMagikarp, TheNitromeFan, cloneembedreportprint, petertodd. In their LessWrong write-up, they showed that asking a model to simply repeat these strings produced evasion, insults or substitutions. ChatGPT, for example, repeated " SolidGoldMagikarp" as "distribute." Some of these tokens even made GPT-3 give different answers at temperature 0, which is supposed to be deterministic.

Their partial explanation goes back to the point about merges. The tokenizer was trained on scraped web text that included Reddit usernames, e-commerce site internals and game logs. The model was trained on more heavily curated text. So strings like SolidGoldMagikarp (I confirmed it's a single token, #43453, in GPT-2's vocabulary) earned a vocabulary slot but almost never showed up during model training. Their embedding vectors stayed close to wherever they started, and the model doesn't know what to do when it meets one.

It's not just a GPT-2 curiosity. Fishing for Magikarp (Land and Bartolo, 2024) built automatic detectors that combine tokenizer analysis, weight-based signals and prompting, and found under-trained tokens across a wide range of models. In newer OpenAI vocabularies, " SolidGoldMagikarp" is 5 ordinary tokens (·Solid | Gold | Mag | ik | arp), but each new tokenizer can bring its own oddities. If you train or fine-tune models, scanning for under-trained tokens is cheap insurance.

The Other Tokenizer Families

BPE dominates today's LLMs, but you'll run into two other algorithms, often through one library:

WordPiece (developed by Google for BERT) also learns merges bottom-up, but according to the Hugging Face course it scores a pair as its frequency divided by the product of its parts' frequencies, which favors pairs whose parts are rare on their own. It keeps only the final vocabulary and encodes each word by taking the longest piece it knows, then the next. Pieces inside a word carry a ## prefix, as in token ##ization. (Google never open-sourced its training code, so the course describes a best-guess reconstruction.)
Unigram LM, from Taku Kudo's Subword Regularization (2018), works top-down. It starts with a large candidate vocabulary, scores how much each piece contributes to a probabilistic model of the corpus, and prunes the least useful pieces. Since the same word can be segmented several ways, Kudo trained translation models on sampled segmentations as a kind of noise, which made them more robust, especially in low-resource and out-of-domain settings.
SentencePiece (Kudo and Richardson, 2018) is a library rather than an algorithm. It implements both BPE and Unigram and trains directly on raw sentences, with no language-specific pre-splitting. Whitespace becomes the visible symbol ▁, so I saw a girl becomes ▁I ▁saw ▁a ▁girl (example from the SentencePiece README). That makes it a natural fit for languages that don't put spaces between words, like Japanese and Chinese.

Picking a Vocabulary Size

Why not a million tokens? Every entry costs a row in the input embedding matrix and another in the output layer that scores the next token (unless the two are tied). With 4,096-dimensional vectors, a 200,000-entry vocabulary is about 820 million parameters per matrix. More entries also means fewer training examples per rare token, which is the glitch-token problem again.

Smaller vocabularies mean longer sequences: attention costs more, the context window fills faster and generation takes more steps. Recent models have moved toward larger vocabularies (50K, then 100K, 128K and 200K) as the embedding table becomes a small share of total parameters. Meta credits Llama 3's 128K tokenizer with "substantially improved model performance."

A Cheat Sheet for Engineers

Common tokenizer gotchas, their causes and practical fixes

A few of these deserve a sentence more.

Count tokens with the real tokenizer. Use tiktoken.encoding_for_model("gpt-4o") for OpenAI models, or your provider's token-counting endpoint, before you enforce context limits or estimate costs. Character-based estimates are wrong for code, non-English text and JSON.

Watch the boundary at the end of your prompt. Because GPT-style tokenizers attach the space to the following word, " Paris" is a single token. A prompt that ends with a trailing space ("The capital of France is ") leaves a bare space token, and the model then has to continue with "Paris" without its usual leading space, an unusual sequence it saw less often in training. Trim trailing whitespace from completion-style prompts. The "token healing" technique cited by Singh and Strouse goes further: it backs up a few tokens and lets the model regenerate across the boundary.

Special tokens are control characters. Strings like <|endoftext|> map to reserved IDs (100257 in cl100k_base, 199999 in o200k_base) that mark document and message boundaries. By default tiktoken refuses to encode them from ordinary text: encode("hi <|endoftext|>") raises an error. Keep that protection on, so pasted text can't forge a document or message boundary.

Key Takeaways

LLMs read integers, not letters. A tokenizer turns text into IDs before the model runs, and anything inside a token, like its spelling, is invisible unless the model memorized it.
BPE builds the vocabulary by merging the most frequent adjacent pairs, and byte-level BPE guarantees that any input can be encoded with no "unknown" token.
Tokenization explains classic failure modes: letter counting, multi-digit arithmetic (75.6% vs 97.8% for GPT-3.5 depending only on commas), higher costs for some languages (up to 15× longer encodings) and glitch tokens.
The fix is usually to change the representation or hand off the work: spell words out, format numbers deliberately, and call code for anything that must be exact.
Measure with the actual tokenizer. "4 characters per token" is a rough English-only average, not a budget.

ASR-induced failure rate: did the speech path break it?

Vijay Kumar Sridharan — Tue, 13 Oct 2026 14:57:31 +0000

Part 1 ended in the failure-review meeting: the ASR team says the transcripts were fine and the NLU misread them, the NLU team says the transcripts were garbage, QA says the test oracle is stale. Everyone is arguing from the same green dashboard, and nobody has an experiment.

This post is the experiment.

Run everything twice

For every semantic test case in your voice suite, execute it twice through the same frozen downstream:

Text control: reference text straight into the NLU/agent. No audio, no ASR.
Audio path: audio through the ASR, then into that same NLU/agent.

Then read the four outcomes — the attribution table from part 1:

Text control	Audio path	Attribution
Pass	Pass	HEALTHY
Pass	Fail	SPEECH_ATTRIBUTABLE
Fail	Fail	DOWNSTREAM_OR_TEST
Fail	Pass	CONTEXT_OR_ORACLE

This is a controlled experiment, not a metric. The text control holds everything downstream constant; the only thing that changes between the two runs is the speech path. When the two runs disagree, the speech path is the difference, and the argument about whose fault it is ends — the table already routed the ticket.

The per-trial verdict rolls up to a suite-level number: ASR-IFR, the ASR-induced failure rate — the fraction of the full paired suite where the text control passes but the audio path fails. In the implementation, each trial contributes 1 if its attribution is SPEECH_ATTRIBUTABLE and 0 otherwise; ASR-IFR is the mean over all trials. The numerator is the speech-attributable count. The denominator is the whole suite.

The denominator choice is the part worth thinking about, because it's doing more work than the numerator.

Why the denominator matters more than the numerator

The numerator is just a count — 142 trials, in the pilot below. Anyone can count failures. The denominator decides what the number means.

ASR-IFR divides by all trials, not by control-passing trials. The alternative — "of the tests that worked, what fraction did speech break" — is also a meaningful number (0.394 / 0.800 ≈ 0.49 in the pilot). But the unconditional rate answers the question operations actually asks: what fraction of my suite is broken by the speech path right now? It stays comparable across runs even when downstream health changes — if the NLU regresses and text-control accuracy drops, a conditional rate would quietly rebase itself and hide the change; the unconditional rate doesn't move unless the speech path moved. That's what makes it gateable: --max-asr-ifr-delta 0.02 against your last approved baseline means something, because the denominator can't shift under you.

And there is a ceiling built into the definition that keeps the metric honest. ASR-IFR can never exceed text-control accuracy — a trial only counts if the control passed. In the pilot, text control is 0.800, so the worst ASR-IFR can ever be is 0.800. When ASR-IFR approaches that ceiling, the reading is unambiguous: nearly every test the downstream could pass is being broken by speech. The metric degrades gracefully instead of exploding — compare that with WER, which cheerfully reports 1.229 when insertions pile up.

The control is the keystone

None of this works without the text control, and the control has to be boring to be useful. In the pilot's reproduced results, text-control accuracy is 0.800 in every condition — clean, 15 dB noise, 5 dB noise, fast speech, telephony, telephony-plus-noise. The downstream never changed, so every bit of movement in ASR-IFR is the speech path. That flat 0.800 is the entire epistemic foundation: without it, you cannot attribute anything, and you're back to arguing in the meeting.

The control also does triage the metric can't. If text-control accuracy drops between releases, that's a downstream or test-oracle problem — the NLU, the agent, or a stale test — and it goes to those owners, not the ASR team. ASR-IFR moving while text control holds flat is a recognizer problem. WER moving while both hold flat is lexical churn: investigate, then ship. Three numbers, three owners, no meeting.

What the numbers look like across conditions

The paired design is implemented in CAFA-IVR (the worked example for this series), and the 360-trial Conformer-CTC pilot ships per-condition reproductions. Sixty trials each:

Condition	WER	ASR-IFR	Text control
Clean	0.115	0.10	0.800
Telephony (8 kHz)	0.272	0.20	0.800
Fast speech (1.15×)	0.412	0.317	0.800
15 dB noise	0.741	0.367	0.800
Telephony + 15 dB noise	1.121	0.667	0.800
5 dB noise	1.229	0.717	0.800
Overall	0.648	0.394	0.800

Read the 5 dB row the way an operator would. WER 1.229 is nearly uninterpretable — worse than 100%, because insertions pile up past the reference length. ASR-IFR 0.717 says something plain: of the 80% of tests the downstream could pass, speech broke almost 90% of them (0.717 / 0.800). Two out of every five tests in the suite, broken by the speech path. One of those two numbers helps you decide whether to ship. It isn't the first one.

And notice what the table separates that WER alone cannot. In the telephony-plus-noise condition, WER 1.121 and ASR-IFR 0.667 move together; in fast speech, WER 0.412 looks moderate while ASR-IFR 0.317 says nearly a third of the suite broke. The relationship between transcription damage and task damage isn't fixed — it depends on which words the recognizer corrupts, which is exactly what part 2's CEER is for. ASR-IFR tells you the speech path broke it; CEER tells you what it broke. You want both numbers, in that order.

Overall: 360 trials, 142 speech-attributable failures, ASR-IFR 0.394. Nearly two in five working tests, broken by speech. That's the release-gate number:

cafa-ivr compare \
  --baseline baseline/summary.csv \
  --candidate candidate/summary.csv \
  --max-asr-ifr-delta 0.02

A candidate that moves ASR-IFR more than two points against baseline doesn't ship until someone explains the movement. WER can move however it likes in the meantime — it's diagnostic, not decisive.

The honest boundary

Stated plainly, as in parts 1 and 2: this is a 360-trial pilot on controlled synthetic speech with a constrained in-domain Conformer-CTC recognizer. It demonstrates the method — the paired design, the attribution table, the gate. It is not a production-ASR benchmark and should not be read as one. Your numbers will differ; the experimental design won't.

The operational takeaway

The question to take back to your team is the one from part 1: for every failing voice test you have right now, can you say whether the speech path caused it? If you run the paired design, the answer comes with a number attached — and the number routes itself. ASR-IFR up, text control flat: fix the recognizer. Text control down: fix the NLU or the test. WER up, both flat: ship.

Three numbers, three owners, no meeting. That's the whole pitch.

Next in the series: CIER — weighting failures by realized consequence, or why "the task failed" is still not the end of the story.

Was My 2,103-Token Tokenizer a Mistake?

Raihan — Tue, 13 Oct 2026 14:30:21 +0000

A compute-matched vocabulary-size study for small code models.

Scope note: 12 LLaMA-style decoders trained from scratch on TypeScript/JavaScript, 3 sizes × 4 vocabulary sizes, equal training FLOPs. Not a scaling law — a controlled experiment.

The problem

I trained ORCH-Fusion (2,103 tokens, 272.7M) and ORCH-Next.js-350M-v2 (16,000 tokens, 287M) on the same backbone but different tokenizers, data, and training. So I cannot say whether the small vocabulary helped. This project fixes that: the tokenizer is the only variable.

The approach

Data: 7,606 exact-hash-deduplicated TS/JS files from HuggingFace
Tokenizers: 4 byte-level BPE tokenizers (2k, 8k, 16k, 32k vocab) trained on the same bytes
Models: 3 LLaMA-style sizes (nominal 10M / 25M / 50M tags; real parameter counts in the Results section)
Training: 1,000 steps each (an under-trained regime, so read the results as vocabulary choice at this budget)
Evaluation: Held-out bits-per-byte on TS/JS code

Fertility table

Vocab size	Tokens per byte
2,000	0.4475
8,000	0.3667
16,000	0.3435
32,000	0.3275

Larger vocabularies have lower fertility — fewer tokens per byte. But they also have larger embedding tables, which cost more FLOPs. The question is whether the quality gain outweighs the compute cost.

Results

Held-out bits-per-byte on TypeScript/JavaScript, 1,000 steps per run.

Size (non-emb)	2k	8k	16k	32k
10M	1.391	1.168	1.219	1.321
25M	1.378	1.258	1.218	1.280
50M	1.436	1.204	1.147	1.458

(10M rows are means of 2 seeds; seed noise reaches 0.13, so 8k vs 16k at 10M is an honest tie.)

The size tags are nominal. Real parameter counts:

Size tag	Non-embedding params	Total params at 2k / 8k / 16k / 32k vocab
10M	3.9M	5.0M / 8.0M / 12.1M / 20.3M
25M	13.3M	14.8M / 19.4M / 25.6M / 37.9M
50M	31.5M	33.5M / 39.7M / 47.9M / 64.2M

The "10M / 25M / 50M" tags are nominal; the table gives the real counts. All 16 runs use max_steps: 1000 on 7,606 deduplicated TS/JS files, an under-trained regime: conclusions are about vocabulary choice at this budget, not at compute-optimal training.

Key findings

8k–16k wins at every size. 2k is always bad; 32k collapses at 50M (its embedding matrix is about half the model and is under-trained at 1,000 steps).
Vocabulary beats backbone: the 10M tag with 8k vocab (8.0M total params, 1.168 bpb) beats the 50M tag with 2k vocab (33.5M total params, 1.436 bpb): 4.2× fewer total parameters, 18.7% lower bits-per-byte.
Seed noise is large (up to 0.13 between the two 10M seeds) — differences below ~0.1 are ties, reported as such.
Was the 2,103-token tokenizer a mistake? Directionally yes — 2k costs ~19% bpb vs 8k at 10M. Caveat: ORCH differs in data and training too, so this is suggestive, not conclusive.

Limitations

3 sizes is fragile for fitting a scaling law
Differences may sit inside seed noise (second-seed row shows the band)
At 10-50M params with 32k vocab, embeddings dominate parameter count
The 3060's memory bandwidth limits throughput; measure MFU instead of guessing

What's next

Add frontier API baselines
Test the curve on a larger held-out run
Apply the finding to the next ORCH model

Repo: github.com/raihan-js/vocab-tax · 7 tests green. The tokenizer is the variable.

Day 35: BERT: Architecture and Masked Language Modeling

Priyesh Dave — Tue, 13 Oct 2026 14:08:08 +0000

Previously, on Day 34: Explained the concept of pretraining in NLP, highlighted why it is foundational, introduced transfer learning, and demonstrated how pretrained models are used in practice with Hugging Face Transformers.

What is BERT? Contextualized Language Representations

BERT stands for Bidirectional Encoder Representations from Transformers. It's a language model—a machine learning system trained to read, understand, and generate human language.

Earlier models read text in only one direction: either left to right or right to left. If you asked them to predict a missing word, they could only use the words before or after the blank—not both at once.

BERT changed this with bidirectionality. BERT reads text in both directions at the same time. When BERT tries to understand or predict a word, it looks at the whole context—everything before and after the target word. In natural language processing (NLP), this is called contextualization: the meaning of a word is adjusted based on the words around it.

For example, “bank” in “river bank” is different from “bank” in “savings bank.” Earlier models sometimes missed this. BERT uses the whole sentence to find the right meaning for every word, every time. This context-aware reasoning is a big reason for BERT’s accuracy and popularity.

BERT’s Architecture: Key Building Blocks

BERT is built from a stack of transformer encoder blocks. These blocks process the entire sentence at once, letting every word send and receive information from every other word.

Here’s what goes into BERT processing:

Tokens: BERT doesn’t work with full words. It works with tokens—often pieces of words, called subwords. A tokenizer breaks text into tokens and converts each token to a unique number.
Segments: BERT can process two partitions of text together, such as a question and its answer. Each token gets a label marking which segment it comes from.
Positions: Transformers have no built-in sense of order. So BERT adds a position embedding to each token to indicate where it appears in the sentence.
Encoder blocks & Attention: Each encoder block uses attention. Attention means that, for every token, the model decides how much to “pay attention” to every other token. These attention weights help BERT understand connections, even between far-apart words.

Imagine BERT's input like a grid: each row is a token, labeled with its segment and position, with attention creating links between all pairs of tokens in each layer.

Masked Language Modeling (MLM)

Traditional language models predict the next word in a sentence. BERT uses Masked Language Modeling (MLM), which works differently.

Here’s how MLM training works:

Some tokens in the input are randomly replaced with a special token: [MASK].
BERT doesn’t know what [MASK] means. Its task is to use the rest of the sentence to guess the original word.
For example:
- Input: “The cat [MASK] on the mat.”
- BERT needs to predict that [MASK] should be “sat”.

Why mask random words?

Unlike left-to-right models, BERT uses context from both left and right to make its guess.
This forces BERT to actually understand the whole sentence—where every word matters, not just the next word.

This approach teaches BERT to model words in any position, based on their full context.

A Step-by-Step Example: How BERT Learns

Let’s follow what happens on a single training step:

Consider the real sentence:

“The [MASK] barked loudly in the park.”

Let’s say “[MASK]” covers the word “dog.”

1. Tokenization and Masking

Split into tokens: [“The”, “[MASK]”, “barked”, “loudly”, “in”, “the”, “park”, “.”]
Each token is mapped to a unique integer.

2. Input Assembly

Each token is converted into an embedding (a list of numbers).
Segment and position information are added.

3. Encoding

Embeddings flow through the stack of encoder blocks.
Attention lets tokens (like “[MASK]”) use information from all the other words, both before and after.

4. Mask Prediction

At [MASK], BERT combines the contexts and outputs a guess—a probability for each possible word.
In our example, “dog” should have the highest probability.

5. Loss and Update

BERT compares its guess to the correct word (“dog”).
It computes a loss (a measure of how wrong it was), then tweaks its internal weights to do better next time.

Repeat this process millions of times, and BERT becomes great at filling in blanks and understanding all kinds of language patterns.

Fine-Tuning: Using BERT for Real Tasks

MLM pretraining gives BERT strong general language skills, but practical NLP tasks—like classifying sentences or extracting names—are different.

To make BERT useful for a specific task, we fine-tune it: start with the general pretrained BERT, then adjust (train) it on a small, labeled dataset for your target problem.

The process:

Add a simple new task-specific layer on top (like a classifier).
Train on labeled data for a few more steps.
BERT keeps its language skills; the new layer learns how to use them for your task.

This transfer learning method was a breakthrough. With BERT, almost any NLP task can see big improvements, often with minimal extra training.

Minimal Code Example: Masked Token Prediction with BERT

Here’s how BERT predicts a masked word using the Hugging Face Transformers library:

from transformers import BertTokenizer, BertForMaskedLM
import torch

# Load pre-trained BERT
tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
model = BertForMaskedLM.from_pretrained('bert-base-uncased')

# Example sentence with a masked token
sentence = "The quick brown [MASK] jumps over the lazy dog."
inputs = tokenizer(sentence, return_tensors='pt')

# Find the position of [MASK]
mask_token_index = torch.where(inputs["input_ids"][0] == tokenizer.mask_token_id)[0]

# Pass the sentence through BERT
with torch.no_grad():
    outputs = model(**inputs)
    logits = outputs.logits

# Get BERT's top predictions for [MASK]
mask_token_logits = logits[0, mask_token_index, :]
top_tokens = torch.topk(mask_token_logits, 5, dim=1).indices[0].tolist()

print("Top predictions for [MASK]:")
for token in top_tokens:
    print(tokenizer.decode([token]))

This code masks the word “fox” in a familiar sentence. BERT examines the context on both sides and predicts the missing word. The top guesses almost always include “fox,” showing BERT’s strong understanding of how language works.

Summary

BERT introduced a powerful new approach for machine understanding of language: using bidirectional, contextualized representations built with transformers and attention. By learning to predict masked words using all available context, BERT sets a foundation for most modern NLP systems. Its bidirectional context-awareness is now the standard for how machines should interpret language.

Key Takeaways

BERT reads text bidirectionally, using context from both sides of each word.
Tokenization, segment and position embeddings, and attention enable contextual understanding.
Masked Language Modeling trains BERT to predict masked words using the full sentence context.
Fine-tuning allows BERT to adapt to specific NLP tasks with minimal new training.
BERT's transformer architecture is now foundational for many modern NLP systems.

Try It Yourself

Choose a short sentence (such as 'The sun rises in the east.') and manually replace one word with [MASK]. Use Hugging Face Transformers to encode the sentence and predict the masked token with BERT, as shown in the lesson's code example. Compare the top predictions to your expectations and reflect on how the context affects BERT's choices.

Further Resources

Coming up on Day 36: Fine-Tuning BERT for Downstream Tasks

How to Identify Technologies Mentioned in a YouTube Video

WgeorgeAssistantIA — Mon, 12 Oct 2026 14:41:54 +0000

"Which technologies does this video talk about?" sounds like a named-entity task, and it is, with a twist: the entities are product names that are short, ambiguous, constantly new and often misheard. This post covers the three problems you meet, and a practical approach to each.

Problem 1: what counts as a technology?

If you do not decide, the model will. Without a definition you get everything from "JavaScript" to "the cloud" to the presenter's company. Write the definition into the prompt:

Include: named products, frameworks, libraries, languages, services, CLIs and protocols
Exclude: generic words ("API", "database", "framework"), company names that are not products, channel names, people

Then maintain a block list of the noise you actually observe. In our runs, recurring false positives included generic words such as "API" and "Google", and the names of channels.

Problem 2: the same tool, many spellings

Speakers say "Postgres", "PostgreSQL", "Postgres DB". Captions write "Next JS", "next.js" or "Nexjs". Two things help:

Normalise: lower-case, remove spaces and punctuation, strip common suffixes (js, db) to create a matching key
Keep an alias list for the display name and for deduplication

Watch for duplicates that differ by one letter. We saw a tool appear twice because one occurrence was misspelt; a fuzzy match on names within the same video catches it.

Translation can also distort names. If your pipeline summarises in another language, instruct the model to keep product names exactly as spoken. A product name turned into a translated phrase is no longer searchable.

Problem 3: mentioned does not mean recommended

A technology can be:

Used in the demo
Recommended as an alternative
Compared against another
Criticised or dismissed

Classify the mention type, or at least keep the quote so a reader can judge. This is the difference between a useful tool list and a misleading one.

Ambiguous names need context

"Neon", "Serena" and "Phantom" are all valid product names, and all share their name with something unrelated. Give the model the surrounding sentence, and when you look up the official site, pass the category as well. A good result for an ambiguous name needs context at both extraction and resolution.

Test it on real videos

Pick ten videos in your field, run the pipeline and read the list yourself. You will quickly see what kind of noise appears and where the prompt needs a rule. Update the block list and the prompt, run again.

Our implementation

We built these rules into VidScope, which returns the tools mentioned in a YouTube video with a timestamp and an official link, and avoids returning a link when the match is unclear.

FAQ

Is this a named-entity recognition problem?
Yes, with the twist that names are short, ambiguous and often misheard in captions.

How do I stop generic words being returned?
Define what counts as a tool in the prompt and keep a block list of observed noise.

How do I handle different spellings of the same tool?
Normalise names to a matching key and keep an alias list.

Evaluate AI Dialogue Rewrites by Tactic, Not Just Fluency

shizhiliang — Mon, 12 Oct 2026 00:02:29 +0000

A dialogue generator can produce five polished alternatives that all do the same thing: ask again, with different adjectives. If your writing tool labels these “more dramatic,” fluency has become a proxy for a change it never actually measured.

Here is a small, human-reviewed evaluation workflow for separating a wording change from a tactical change. It is a proposed test design, not a benchmark result or a claim about any product's implementation.

Start with an observable distinction

Use an invented scene. Mara needs her coworker Eli to sign a transfer request before the office closes. Eli wants to know who will take over his unfinished work.

Original line: “Please sign it before five.”

Candidate A: “I really need your signature before five.”

Candidate B: “If I take the Friday handover, will you sign it now?”

Candidate C: “The director already approved it. You are only holding everyone up.”

A intensifies the request. B offers a trade. C invokes authority and applies pressure—but invents an approval unless that fact already exists in the scene. A tactical change is not automatically an acceptable rewrite.

1. Make the fixture small enough to inspect

Keep the source exchange, known facts, and requested change together. For example:

{
  "goal": "Mara wants Eli to sign before five",
  "known_facts": [
    "Eli is worried about his unfinished work",
    "Mara can offer to handle Friday's handover"
  ],
  "unknown_facts": ["Whether the director approved the transfer"],
  "source_tactic": "direct request",
  "requested_tactic": "offer a concrete trade",
  "must_preserve": ["Eli has not agreed yet"]
}

Do not turn missing information into false certainty. “Unknown” means the evaluator should not accept a new claim as established fact; it does not mean the claim is necessarily false.

2. Ask for evidence, not a quality score

A useful review record has four fields:

Observed tactic: What is the speaker trying to get the other person to do, and how?
Evidence: Quote the exact words that support that interpretation.
New commitments or facts: What did the rewrite add?
Decision: Accept, revise, or uncertain—with one reason.

For B, the evidence is “If I take the Friday handover.” The added commitment is explicit and supported by the fixture. It is a reasonable candidate to keep, not proof of a good scene. Eli's response still has to make dramatic sense.

For C, the authority tactic is visible, but the approval is unsupported. Flag that separately rather than rewarding the stronger tone.

3. Test the evaluator with contrast pairs

Before relying on a reviewer model, include pairs where a fluent answer should not win:

Same request, more emotional wording.
A genuine trade versus an unrelated new offer.
A change of tactic that preserves facts versus one that invents leverage.
An indirect line with two plausible readings, where “uncertain” is appropriate.

Have a person label the pairs first. Compare the model's evidence and decisions against those judgments. If people disagree, keep the disagreement visible; do not hide it behind a single reference label.

This is a qualitative test set. With only a few examples, a pass percentage can create more confidence than the evidence deserves. Log concrete failure modes instead: unsupported leverage, paraphrase mistaken for strategy, or ambiguous subtext treated as certain.

Where automation should stop

Simple code can check whether required fields are present. It cannot establish that a character's offer feels honest, surprising, or dramatically appropriate. A second model can suggest a reading, but it is not independent proof that the first model improved the scene.

The final review question belongs to the writer: does the other character now have a different reason to answer?

Disclosure: I work on Laper. This article was generated with AI for our editorial account. The fixture and evaluation workflow above are illustrative, not a description of a shipped Laper feature or measured results.

Try the test on one exchange before expanding it into a pipeline. If every alternative gives the listener the same choice, you may have generated variety in wording—not variety in action.

Building a Sentiment Analysis Model with Python and Hugging Face Transformers

Phil IPS — Sun, 11 Oct 2026 22:33:00 +0000

Building a Sentiment Analysis Model with Python and Hugging Face Transformers

Sentiment analysis is a crucial technique for understanding the emotional tone behind text. In this tutorial, we'll build a simple yet powerful sentiment analysis model using Python and Hugging Face's Transformers library.

Step 1: Set Up Your Environment

First, install the required libraries:

pip install transformers torch

Step 2: Load a Pre-trained Model

We'll use a pre-trained model fine-tuned on the SST-2 dataset for sentiment analysis:

from transformers import pipeline

# Initialize the sentiment analysis pipeline
classifier = pipeline("sentiment-analysis")

Step 3: Analyze Sentiment

Now, let's analyze some sample texts:

# Example texts to analyze
texts = [
    "I love this product! It works perfectly.",
    "This is the worst experience I've ever had.",
    "The movie was average, nothing special."
]

# Run sentiment analysis
for text in texts:
    result = classifier(text)
    print(f"Text: {text}")
    print(f"Sentiment: {result[0]['label']} (Confidence: {result[0]['score']:.2f})")
    print()

Step 4: Customize for Your Use Case

You can further customize the model by fine-tuning it on your own dataset. For beginners, the above setup is sufficient for basic sentiment analysis tasks.

Conclusion

In this tutorial, we've demonstrated how to build a sentiment analysis model using just a few lines of Python code. With the power of Hugging Face Transformers, you can easily integrate sentiment analysis into your applications, from customer feedback analysis to social media monitoring.

Feel free to experiment with different models and datasets to improve accuracy or adapt to specific domains!

معالجة اللغة العربية الطبيعية: لماذا تربك اللهجات النماذج؟

aryamkhald — Sun, 11 Oct 2026 19:09:43 +0000

العربية لغة واحدة على الورق، وعشرات اللهجات على ألسنة الناس. ونظام يحلل تقييمات العملاء يصطدم بهذا فوراً.

أين تظهر الصعوبة

التشكيل والإملاء: "الى" و"إلى"، و"ة" و"ه" في نهاية الكلمة. كتابة الناس لا تلتزم بقاعدة.

اللهجات: "وش"، "إيش"، "شو"، "ايش" كلها بمعنى "ماذا". والنموذج المدرَّب على الفصحى قد لا يفهمها.

العربيزي: كتابة العربية بحروف لاتينية وأرقام: "7abibi".

السخرية: "ممتاز، تأخر الطلب أسبوعاً!" نص إيجابي ظاهرياً.

الصيغ الصرفية: الكلمة الواحدة تحمل ضميراً وحرف جر وفعلاً: "فسيكتبونها".

كيف نتعامل معها

تطبيع النص: توحيد أشكال الألف والياء وإزالة التطويل.
بيانات مصنّفة محلياً: عيّنات من تعليقات حقيقية بلهجة منطقتك.
تقييم منفصل لكل لهجة حتى لا يخفي المتوسط ضعف الأداء في إحداها.
مراجعة بشرية للحالات منخفضة الثقة.

مثال على التطبيع

import re

def normalize(text):
    text = re.sub("[إأآ]", "ا", text)
    text = re.sub("ـ+", "", text)
    text = re.sub("[\u064B-\u0652]", "", text)
    return text.strip()

الخلاصة

لا تقبل مؤشر دقة واحداً يعلنه المورّد. اطلب اختباراً على بياناتك أنت. نعمل في هذا المجال ضمن معالجة اللغات الطبيعية.

How to Add Chinese NLP to Your AI Agent in 5 Minutes

leonmch — Sun, 11 Oct 2026 11:42:17 +0000

Most LLMs read Chinese well at the sentence level. What they handle badly is the
mechanical layer underneath: cutting a paragraph into words the way a search
index would, romanising names so they sort correctly, pulling keywords with
weights, or scanning text against a list you maintain yourself.

Those four operations are cheap to run locally. I packaged them as an MCP server.

The problem

Chinese has no spaces between words, so "我喜欢北京天安门" has to be cut
somewhere, and where you cut it changes what everything downstream sees. Segment
it wrong and your keyword extractor returns fragments, your index misses
matches, your slug generator emits gibberish.

LLMs guess at this. Ask one to split a paragraph into words and you get something
plausible that does not match any tokenizer on your stack. Ask it to convert
names to pinyin for sorting and it will be right most of the time, which is
harder to catch than a consistent error.

The usual fix is a cloud API. That works, and it means every document your agent
processes leaves your machine, plus a latency dependency and a cost line.

The solution

chinese-nlp-mcp exposes four working tools over stdio, all in-process:

segment_chinese — jieba segmentation, default / search / index modes
convert_pinyin — pypinyin, tone / tone2 / initials / first_letter
extract_keywords — TF-IDF extraction with weights
detect_sensitive_words — Aho-Corasick against a caller-supplied word list

Plus hello_world for health checks.

Install

uvx chinese-nlp-mcp

Or with pip:

pip install chinese-nlp-mcp
chinese-nlp-mcp

Python 3.10+. Also on PyPI and the Official MCP Registry as
io.github.leonmch-byte/chinese-nlp-mcp.

Setup

Claude Desktop (claude_desktop_config.json):

{
  "mcpServers": {
    "chinese-nlp-mcp": {
      "command": "uvx",
      "args": ["chinese-nlp-mcp"]
    }
  }
}

Cursor (.cursor/mcp.json):

{
  "mcpServers": {
    "chinese-nlp-mcp": {
      "command": "uvx",
      "args": ["chinese-nlp-mcp"]
    }
  }
}

VS Code (.vscode/mcp.json) uses servers, not mcpServers:

{
  "servers": {
    "chinese-nlp-mcp": {
      "type": "stdio",
      "command": "uvx",
      "args": ["chinese-nlp-mcp"]
    }
  }
}

Demo

Four product reviews, the kind you get pasted into a ticket:

物流很快，包装完好，质量不错。
客服态度很好，解决问题很耐心。
发货太慢了，等了五天才到。
质量太差了，收到就是坏的，申请退款。

On its own this is fine for an LLM to summarise. But if you want to route it,
group it, or track themes over time, you need keywords first. Ask Claude to call
extract_keywords:

质量   0.610
发货   0.536
退款   0.531
太慢   0.510
太差   0.500
客服   0.456
五天   0.450
完好   0.431

Weights are raw TF-IDF, unnormalised, so they exceed 1.0 on longer documents.
质量 and 发货 are the themes; 退款, 太慢, 太差 are the complaints. That
is a routing decision the LLM can now make on numbers instead of vibes.

Two more calls make it useful. Scan against your own escalation terms:

detect_sensitive_words(text=reviews, words=["退款", "太差", "坏的"])
→ {'matches': [{'word': '太差', 'index': 48},
               {'word': '坏的', 'index': 56},
               {'word': '退款', 'index': 61}], 'clean': False}

And segment when you need offsets rather than topics:

segment_chinese("物流很快，包装完好，质量不错。", mode="search")
→ ['物流', '很快', '，', '包装', '完好', '，', '质量', '不错', '。']

(jieba keeps punctuation as its own token; filter it out if you don't want it.)

One gotcha worth knowing: initials and first_letter are both per
character. 中国 is zhong guo, so you get zh g and z g, not zh and z.
Take [0] yourself if you want one letter per word.

Free vs Pro

The free tier is 500 segment_chinese, 300 convert_pinyin, 100
extract_keywords and 200 detect_sensitive_words calls per day, resetting at
00:00 UTC. That covers individual use comfortably. hello_world is never
throttled, so a health check can never look like a broken server. Hitting a limit
returns an error naming the limit, the reset time and what Pro adds.

Pro is $19 once, not a subscription. It adds batch processing (100 texts per
call), custom dictionaries and priority support. Batch is the one that matters
if you are running this over a corpus rather than a ticket queue — without it
you pay one round trip per document, and the per-call overhead dominates on
short texts. Custom dictionaries matter if your domain has vocabulary jieba has
never seen, which for most Chinese technical corpora it has.
Ready to upgrade? Get Pro on Gumroad → — $19 one-time, no subscription.

Privacy

Without a license key: fully offline. No network calls, no telemetry. Your text never leaves the machine.

With a Pro license key: the key is verified against Gumroad once at startup, cached for 24 hours. Only product_id and license_key are sent — no text content, no usage statistics, no machine identifiers. You can turn verification off entirely by not setting CHINESE_NLP_LICENSE_KEY.

Either way the tools themselves never make a network call. FastMCP's update check is disabled at startup; jieba, pypinyin and the automaton all run in-process.

Daily counters live in a local file (%LOCALAPPDATA% on Windows, ~/Library/Application Support on macOS, $XDG_STATE_HOME on Linux), signed with HMAC-SHA256. That file holds only a truncated hash of your key, never the key itself, unless you explicitly opt in.

Two things we state plainly. First, the daily counters are tamper-resistant but not tamper-proof — this is MIT-licensed local software and anyone determined enough can patch it. Second, there is no DRM. Device binding records a hashed machine ID and warns past three devices, but never blocks. Losing a little revenue beats locking out someone who paid.

Try it

https://github.com/leonmch-byte/chinese-nlp-mcp

Segmentation bugs are the interesting reports. If jieba gets a name, a jargon
term or a domain term wrong, that is the bug I want to hear about. So is
feedback on whether search should be the default mode, and on whether
initials needs a word-level variant.

DEV Community: nlp

Day 36: Fine-Tuning BERT for Downstream Tasks

Why Fine-Tune BERT?

Understanding Downstream Tasks

How Fine-Tuning Works: High-Level Steps

Fine-Tuning BERT for Text Classification: A Minimal Example

Example Code

Avoiding Common Fine-Tuning Pitfalls

Key Takeaways

Try It Yourself

Further Resources

I Thought Extracting Dates Was Easy Until I Built an On-Device iOS App

Finding a date isn't the same as understanding it

Starting with Turkish

Why I chose on-device processing

Thinking about Apple's evolving AI models

A small problem, but an interesting one

What Changes When AI Text Is Humanized: 138 Pairs, Measured

The one question nobody has measured

Method

What changed

Composite AI-tell density — the one that moves

Cliché phrases — the single most-moved measure in the study

Slop vocabulary

Sentence rhythm (coefficient of variation) — up is the intended direction

Length — humanized text comes back slightly longer

Paragraph structure — never touched, 138 of 138

What barely changed, or got worse

What this does and does not say

How to use it

Method appendix

Why Can't LLMs Count the R's in "Strawberry"? How Tokenizers Work

Tokens: the Model's Only Alphabet

From Text to Token IDs in Five Steps

How BPE Learns Its Vocabulary

Byte-Level BPE: No More "Unknown Word"

Why the Model Can't See the R's

The Multilingual Token Tax

Numbers: Why Long Addition Goes Wrong

Glitch Tokens: When the Vocabulary Outlives the Data

The Other Tokenizer Families

Picking a Vocabulary Size

A Cheat Sheet for Engineers

Key Takeaways

Further Reading (sources used)

ASR-induced failure rate: did the speech path break it?

Run everything twice

Why the denominator matters more than the numerator

The control is the keystone

What the numbers look like across conditions

The honest boundary

The operational takeaway

Was My 2,103-Token Tokenizer a Mistake?

The problem

The approach

Fertility table

Results

Key findings

Limitations

What's next

Day 35: BERT: Architecture and Masked Language Modeling

What is BERT? Contextualized Language Representations

BERT’s Architecture: Key Building Blocks

Masked Language Modeling (MLM)

A Step-by-Step Example: How BERT Learns

Fine-Tuning: Using BERT for Real Tasks

Minimal Code Example: Masked Token Prediction with BERT

Summary

Key Takeaways

Try It Yourself

Further Resources

How to Identify Technologies Mentioned in a YouTube Video

Problem 1: what counts as a technology?

Problem 2: the same tool, many spellings

Problem 3: mentioned does not mean recommended

Ambiguous names need context

Test it on real videos

Our implementation

FAQ

Evaluate AI Dialogue Rewrites by Tactic, Not Just Fluency