(This is a translated/improved version of the french article Raku en 2020 from the same author)
Thank you to Elizabeth Mattijsen who helped to proofread and with advices.
Table Of Contents
- Birth
- Renaming process from Perl 6 to Raku
- Implementations based on Parrot
- Implementations based on Pugs
- Other implementations
- Conclusion
The history of Raku
The history of Raku interpreter and virtual machine is really not straightforward... If you've heard about "Parrot", "PIR", "PASM", "Pugs", "Rakudo", "MoarVM", "NQP", "Rakudo on JVM", "Rakudo on Parrot", "Mildew", "Ponie" but you're lost with all these implementations then this blog post is made for you!
I made researches 🔍 to clarify all historical and technical (!) details and gathered all the results here, with several homemade and original sketches to make it crystal clear for you.
The birth of Raku
Raku is born on 18 July 2000 from a famous event where John Orwant threw coffee mugs against a wall during a meeting.
The renaming process : Perl 6 → Raku
A programming language that changes its name... it does not happens often!
Recently, Raku celebrated its second year (or its 21 years). Here is how the renaming happened:
- Long time ago (around 2005), people discussed about an alias but it never led to anything
- while answering a question, Larry Wall proposed an alias
- alias requested by some people from Perl 6 community
- and validated by Larry
- later, one member of the core team (Elizabeth Mattijsen) opened an "issue" on the official GitHub of Raku to ask for a renaming:
- Elizabeth then opened a merge request towards the Git official repository of Raku (requesting maintainers to vote):
- people voted for (or abstained), then came the "blessing" from Larry Wall:
- renaming done, please welcome Raku!
The renaming process in links:
- August 2017: Original idea for Perl 6 Alias
- October 2018: A request to Larry Wall for an alias
- November 2018: On Raku
- August 2019: Liz issue — Perl in the name is confusing
- August 2019: Liz PR — Path to Raku
- September 2019: On renaming Perl 6
- October 2019: Blessing of Larry then renaming
Compilers, frameworks and virtual machines
In this second part, I will detail the past implementations of the Raku language. Get ready, acronyms are coming!
Which "official" implementation?
Short answer: Rakudo and MoarVM.
(the details about Rakudo/MoarVM are in the previous blog post of the serie)
Initially it was decided that no implementation would be defined as "official" for Raku but rather that "Raku is something that passes the official test suite". In fact, since 2012 there is an official de facto implementation which is Rakudo + MoarVM.
Before that, there was the Parrot framework (2002-2012) as well as Pugs (2005-2007) and various other compilers and runtime environments (Mildew, Niecza, Yapsi…). This technical and historical part will be discussed at the end of this dispatch. But let's go back to talk about the beginnings.
The very first announcement of the creation of Raku dates back to July 19, 2000. Larry Wall's idea was to make it an ultra-community project right down to the language design, so he would start by asking to contribute to the specifications. It resulted in 361 RFCs, , from which Larry Wall produced apocalypses, which where then explained by Damian Conway in the exegesis , and then later further detailed in the synopses.
Links for historians or the curious:
- 2000-1018: the Raku museum repository partially saves "events" that are part of Raku's history
- 2000: Old blog post criticizing the RFC process (early)
- 2005: Get involved in Pugs
- 2006: Get involved in Perl6
- 2008: Get involved in Perl6
- 2009: Get involved in Perl6
- 2010: Happy 10th anniversary Perl 6 (MUST READ ⚠️)
- 2015: Flavio Glock interview
Parrot
Created in 2002, the framework and the Parrot virtual machine have long been central to Raku's implementation (this is no longer the case today). Parrot was born from an April Fool's Day announcing the unification of Perl and Python and even the upcoming release of a book on the subject:
Why build a new VM?
The JVM was studied in 2001, but considered poorly suited to dynamic languages (it still wasn't in 2010, quid of 2021?).
Compilation
Parrot offers much more than a virtual machine, Parrot is a language creation, compilation and execution framework. This framework makes it possible to build high level languages (HLL) easily and quickly. It has been put into practice by Rakudo for a long time.
PCT
PCT stands for Parrot Compiler Tools. Its purpose is to enable the creation of a compiler and a high level language execution environment (HLL).
PCT is written in PIR (see assembly section below).
PAST
PAST stands for Parrot Abstract Syntax Tree: internal representation of a syntax tree.
HLL
HLL stands for High Level Language.
PGE
Originally called P6GE, PGE stands for Perl Grammar Engine. Based on HLLCompiler. A PGE rule looks like this:
rule term { <number> | \( <expr> \) }
rule number { \d+ }
rule expr { <term> ( '+' <term> )* }
PGE was used by Pugs, another Raku compiler (see below).
TGE
TGE stands for Tree Grammar Engine.
PACT
PACT stands for Parrot Alternate Compiler Toolkit. An alternative to PCT because PCT was considered too limited.
Assembly
PASM
PASM stands for Parrot assembly language (PASM). The low level code ready to be converted and executed by Parrot VM. PASM code looks like this:
set I1, 1
REDO:
gt I1, 10, END
print I1
print " "
inc I1
branch REDO
END:
print "\n"
end
References:
- PASM code examples
- LinuxMag articles from french mongueurs on Parrot assembler writing: 1, 2 and 3.
IMCC
IMCC stands for Intermediate Code Compiler. It is an alternative tool for creating and running Parrot bytecode
IMCC brings its own language, commonly known as Parrot Intermediate Language (PIR). IMCC embeds the Parrot runtime environment, so IMCC can compile from PIR to PASM, then from PASM to PBC, then execute this bytecode. IMCC can also do optimizations, although it doesn't by default.
PIR
First called IMC, PIR is an overlay of PASM without being a language of high level. Files containing PIR code have the suffix ".imc"
This is what PIR looks like:
.sub loopy
.local int counter
counter = 0
LOOP: if counter > 10 goto DONE
print counter
print " "
inc counter
goto LOOP
DONE:
print "\n"
end
.end
You can learn more from this IMCC intro or this article on how to write PIR. See here PIR examples or even the detailed documentation of FR mongueurs on PIR.
Execution
NCI
NCI stands for Native Call Interface.
PMC
PMC stands for PolyMorphic Container or Parrot Magic Cookies (the way of representing data in the virtual machine).
PBC
PBC stands for Parrot Byte Code
The Parrot VM
The Parrot VM is a registry-based virtual machine, which is not the "standard" (eg JVM).
Here are some features of the Parrot VM:
- able to manage static and dynamic languages
- Copy On Write
- manages continuations and closures
- can be embedded in C code
- PolyMorphic Container (flexible type storage)
It's easy to include Parrot in C code or call C in Parrot (Native Call Interface).
Raku implementation based on Parrot (2002-2012)
Or with a merged IMCC and Parrot stage (later because IMCC contains Parrot):
Below is an even more detailed/technical diagram (taken from Parrot documentation):
Many details are available in the Parrot Design Document
Problems with Parrot
If Parrot was initially at the center of Raku, this has changed over time:
- 2008: "While Parrot is the most solid solution to deploy Perl 6 in the long term, on the other hand, the challenges Parrot has accepted had proven to consume more time and resources than previously expected." (Perl foundation news)
- 2013: "The latest news is that Parrot Virtual Machine is no longer the only one enjoying Rakudo's exclusivity." (all about Perl6)
- communication problems
- Parrot gradually moved away from Perl 6 (by becoming more generalist)
Little by little, the Rakudo compiler has freed itself from Parrot and is now targeting a new virtual machine.
Pugs (2005-2007)
From here we start archeology, the real one 🔍
In his short existence, Pugs explored many architectural choices. Pugs refers to the interpreter, compiler, runtime environment, and test suite. The compiler can compile to Haskell, JavaScript, Perl 5, or PIR (for Parrot, if you remember).
Pugs is written in Haskell and falls dormant after 2007.
Compilation
Here are the concepts of builds around the Pugs implementation.
PCR
PCR stands for Pugs Compiler Rules. This is a Perl 5 implementation of Raku's regular expression engine. PCR replaced PGE in Pugs.
LREP
LREP evolved into Pugs::Compiler, then, later, MiniPerl6.
Pugs::Compiler
Pugs::Compiler is a set of Perl 5 modules for compiling Raku.
MiniPerl6
Was part of Pugs but in a separate directory. MiniPerl6 became KindaPerl6 then Perlito6.
KindaPerl6
KindaPerl6 or KP6 is a Raku implementation of the Raku grammar with a bootstrap in Perl 5. KindaPerl6 is built from MiniPerl6.
Execution
PIL
PIL stands for Pugs Intermediate Language. PIL is not a human readable format but rather an abstract syntax tree internal to Pugs.
PIL-Run
PIL-Run was built on top of a set of Perl 5 modules that read and run PIL. PIL-Run was Pugs' back-end in Perl 5... at the start.
Pugs based implementations
Pugs + various back‑ends
"Pugs compiles and executes"
"Pugs compiles for Parrot (PIR)"
The compilation to PIR is done in two stages, small zoom on it:
"Pugs compiles (transpiles?) To JavaScript"
"Perl compiles (transpiles?) To Perl 5"
v6
v6-pugs then v6-alpha and now v6.pm is a complete rewrite of Pugs, using a mix of Raku (Pugs::Compiler and associated modules) and Perl 5. It reuses code from the runtime environment of PIL-Run and "half of the CPAN" 😀
Now v6 is released in Perlito6. v6.pm is the front-end for Perlito and Pugs in Perl 5.
Perlito
The Perlito project is led by Flavio S. Glock. The Perlito project actually contains multiple compilers for multiple languages in multiple languages targeting various back‑ends:
- compile Perl 5 to Java source code
- run Perl5 directly in the JVM
- compile Perl 5 to JavaScript source code, run Perl5 directly in the browser or Node.js
- compile Raku to JavaScript source code, run Perl6 directly in the browser or Node.js
- compile from Perl 5 to Perl 5
- compile from Raku to Perl 5
- compile from Raku to Python 2.6
- compile from Perl 5 to Perl 6 (in progress)
- compile from Raku to Ruby 1.9 (in progress)
- compile from Raku to Go (in progress)
- compile from Raku to Common Lisp (SBCL) (in progress)
One thing he can't do is translate from Perl 5 to Raku.
MiniPerl6 and KindaPerl6 form Perlito6. Perlito5 is a port of Perlito6 to Perl5. It is "Perl 5 implemented in Perl 5". Perlito5 itself is written in Perl. Perlito6, on the other hand, is written in Raku.
Perlito only implements a subset of Raku called the "useful subset" which:
- has no list context
- has no inheritance
- has no laziness
- has closures but no coroutines or continuations
- has no multis.
This "useful subset" is kind of similar to NQP (see previous blog post for details about NQP).
Other compilers, runtime environments and virtual machines
In this part, I will continue my tour of initiatives around compilers, runtime environments, and legacy virtual machines.
STD and viv
STD and viv are works of Larry Wall.
We find in STD the grammar of Raku… written in Raku (which did not have a compiler at the time! 😏 It's the beginning of the bootstrapping problems 😀
VIV or literally in Roman numerals "VI → V" (and therefore "6 to 5") was a module capable of converting from Perl 6 to Perl 5.
SMOP
First called YAP6, SMOP stands for Simple Meta Object Programming
SMOP is a compiler and runtime environment written in C for Raku, but we sometimes say SMOP to refer only to the runtime environment (which may be targeted by other compilers).
SMOP is not a virtual machine but a runtime environment that resembles that of Perl 5, while having functionality to manage Raku.
Mildew
Mildew was an STD to SMOP compiler.
It directly uses the STD grammar.
Elf
Elf was a compiler written in Raku with a Ruby grammar. It could issue Perl 5 or Lisp. It was never completed. It was expected that Elf could target SMOP.
Ponie
Ponie stands for Perl On a New Internal Engine. This involved running Perl 5 in the Parrot virtual machine. Stopped in 2006.
Punie
Punie (direct reference to Ponie) was a Perl 1 compiler to Parrot.
Yapsi
Yapsi for Yet Another Perl Six Implementation is an implementation of the compiler and its Raku runtime environments and virtual machines in Raku.
SIC means (S??) Instruction Code, this is Yapsi's own byte code.
Niecza
Niecza is a Raku compiler written in C# that targets CLR (Mono).
Topaz
Topaz was a Perl 5 core rewrite in C++ started before Perl 6, finally abandoned.
Sapphire
Sapphire was another rewrite of Perl 5 internals in September 2000, shortly after the announcement of Perl 6. More experimentation than something else.
Phew! It was the last one, we're done. 😀
Conclusion
There are many interesting concepts to be found in Raku as well as in its history and the history of its implementations (although most of them are part of a vanished world, they well deserved a little closer look).
I hope that this long blog post has been able to lift the fog around the twists and turns of Raku's implementations, its history and to highlight the concepts of the language.
Top comments (2)
I never realize how lost I am, until I read this article.
Great job man.
Thank you 🙏