Machine translation is the oldest continuously funded application of computing to language, the first field to be shut down by an evaluation report, and the field where the current architecture was invented. It also has the most abused numbers in AI, because BLEU scores from different decades are routinely printed in the same table when they cannot be compared at all.
Weaver, 1949: translation as decipherment
Warren Weaver, then at the Rockefeller Foundation, circulated a memorandum titled simply Translation in July 1949, to roughly two hundred people. It is the founding document of the field and its central idea is a cryptographic analogy: Weaver wrote that when he looked at an article in Russian, he could say that it was really written in English but coded in strange symbols, and that he would now proceed to decode.
The analogy is wrong, and the way it is wrong set the field’s agenda for forty years. Encipherment is a bijection over a fixed alphabet; translation is not. But the memorandum also contains ideas that came good much later — Weaver discussed statistical methods, Shannon’s information theory, and the idea that word meaning is determined by context within a window of surrounding words, which is the distributional hypothesis that word embeddings would eventually implement.
The 1954 demonstration and the promise it made
On 7 January 1954, Georgetown University and IBM demonstrated a Russian-to-English translation system on an IBM 701 in New York. The system had a vocabulary of 250 words and six grammar rules, and it translated about sixty carefully chosen sentences, largely on chemistry and general topics.
It worked, on those sentences. The press coverage was extensive and the accompanying claim — associated with Leon Dostert, the Georgetown linguist who led the project — was that translation by machine would be an accomplished fact within three to five years. US government funding followed on that basis.
The pattern here is exactly the one described on the AI winters page: a demonstration on a restricted problem, presented to funders as an early version of the general capability, with a timeline. The sixty sentences were selected in advance and the vocabulary was built for them.
ALPAC, 1966, and the twenty-year gap
Twelve years and a substantial amount of money later, the Automatic Language Processing Advisory Committee reported to the US National Academy of Sciences. Its findings were that machine translation output required so much post-editing that the total process was slower and more expensive than human translation, that there was no shortage of human translators, and that the money would be better spent on basic computational linguistics.
American MT funding was cut sharply and remained low for about two decades. Work continued elsewhere: Systran, founded in 1968, supplied rule-based translation to the US Air Force and later to the European Commission, and — via AltaVista’s Babel Fish from 1997 — was the first machine translation most people ever used. Canada’s Météo system, in production from the late 1970s translating weather bulletins between English and French, is the era’s genuine success and demonstrates the principle that has held ever since: restrict the domain enough and the problem becomes tractable with the methods you already have.
The statistical turn, 1988 to 1993
The break came from IBM’s speech recognition group at Yorktown Heights, which approached translation as a noisy channel problem — the same framing that had worked for speech. Peter Brown, Stephen and Vincent Della Pietra, Robert Mercer and colleagues published the series now known as IBM Models 1 to 5, culminating in The Mathematics of Statistical Machine Translation: Parameter Estimation in Computational Linguistics in 1993.
The enabling resource was a large parallel corpus: the proceedings of the Canadian Parliament, published in both official languages. The method learned word alignments from that corpus and required no linguist to write a single rule. The reaction from the linguistics side of the field was hostile, and the argument that followed is a smaller version of the symbolic-versus-statistical argument running through the rest of AI.
Statistical MT then dominated for two decades: phrase-based models replaced word-based ones, and the open-source Moses toolkit from 2007 made the approach the default. Google Translate launched a statistical system in the mid-2000s and it is what most people meant by machine translation until 2016.
Why you cannot compare BLEU scores across eras
BLEU was introduced by Kishore Papineni and colleagues at IBM in a 2002 paper, BLEU: a Method for Automatic Evaluation of Machine Translation. It compares a system’s output to one or more reference translations by counting matching n-grams, with a brevity penalty to stop systems gaming it by producing short output. It made the field’s progress measurable without a human in the loop, which is why it took over.
It is also the source of the most common piece of misinformation in MT history writing. A BLEU score is not a property of a system. It is a property of a system, a test set, a tokenisation, a number of reference translations and a specific implementation of the metric. Change any of those and the number moves by several points.
- Test set. A score on WMT 2014 English–German has no relationship to a score on a 1990s in-house corpus. Different domains, different sentence lengths, different difficulty.
- Number of references. BLEU with four reference translations scores far higher than BLEU with one, because there are four chances for each n-gram to match. Some older results used four.
- Tokenisation. Whether punctuation is split, whether case is preserved, and how compounds are handled all change the n-gram counts. This is why sacreBLEU exists — a 2018 tool whose entire purpose is to fix the tokenisation and report a signature string so that two numbers can be legitimately compared.
- Language pair. BLEU on English–French and BLEU on English–Finnish are not the same scale in any sense. Morphologically rich languages score lower for the same translation quality, because there are more ways to be correct.
So this page does not print a table of BLEU by decade, and any page that does is showing you numbers that cannot be lined up. What can honestly be said is what each paper reported against its own contemporaneous baseline on its own test set, which is a real comparison because the authors ran both.
The general failure mode here is not specific to translation. It is reporting a metric without the conditions that make it meaningful, and it is the same problem covered in how to read a benchmark result. If a historical number is quoted without its test set, treat it as decoration.
Neural machine translation, 2014 to 2017
- 2013–2014. Kalchbrenner and Blunsom, and then Sutskever, Vinyals and Le in Sequence to Sequence Learning with Neural Networks, encode a whole sentence into a fixed-length vector with one recurrent network and decode it with another. The 2014 paper reports 34.8 BLEU on the WMT’14 English-to-French task for an ensemble of LSTMs against a 33.3 BLEU phrase-based baseline that the authors ran themselves, and 36.5 when the neural model was used to rescore that baseline’s candidate list. Both numbers are on the same test set as the baseline, which is what makes them meaningful.
- 2014–2015. Bahdanau, Cho and Bengio identify the fixed-length vector as the bottleneck — a long sentence has to fit through the same-sized hole as a short one — and introduce attention, letting the decoder look back at all encoder positions with learned weights. This is the mechanism the entire subsequent history of the field is built on.
- September 2016. Google publishes its neural machine translation system and reports, on side-by-side human evaluation against its own previous phrase-based system, an average reduction in translation errors of about 60 per cent across the language pairs tested. This is a human evaluation against a stated baseline, which is a stronger form of evidence than a BLEU delta.
- June 2017. The transformer paper removes recurrence entirely, reporting 28.4 BLEU on WMT 2014 English-to-German and 41.8 on English-to-French, at a fraction of the training cost of the systems it beat. Translation is the application the transformer was built for; everything else it is used for now is a later discovery.
The human parity claims, and what happened to them
In March 2018 Microsoft announced that its system had achieved human parity on translation of news from Chinese to English, on the WMT 2017 test set, based on human evaluation of individual sentences by bilingual annotators.
Two independent groups published re-evaluations the same year, and both found the claim did not hold under stricter conditions. The findings, stated carefully:
- Evaluator expertise matters. When professional translators rather than bilingual crowd workers judged the same output, they preferred the human translations.
- Sentence-level evaluation hides document-level failure. Rated one sentence at a time, machine output looked comparable. Rated as whole documents, human translations were clearly preferred — because consistency of terminology, pronoun reference and discourse coherence across sentences is invisible when you evaluate sentences in isolation.
- Reference quality matters. Where the source text was itself a translation, the direction of translation affected the comparison in ways the original evaluation had not controlled for.
The lasting contribution of that exchange is methodological rather than about translation: document-level evaluation with professional raters became the standard for parity claims, and the field is markedly more careful about them now. The claim was not fraudulent — it was true under its stated protocol, and the protocol turned out to be the wrong one. That is the most common way an AI claim goes wrong, and it is much harder to spot than a fabricated number.
What remains genuinely unsolved is worth listing, because it is where the current systems fail rather than where they failed in 1966: low-resource language pairs where no large parallel corpus exists; gender and other bias inherited from training data, visible when translating from a language without grammatical gender into one with it; terminology consistency across a long document; and the fact that a fluent, confident, wrong translation is harder for a non-speaker to detect than a clumsy correct one.
Top comments (0)