DEV Community: Richard Hainsworth

Enumerating in RakuDoc v2

Richard Hainsworth — Sun, 15 Feb 2026 11:31:14 +0000

We needed a legal document with numbered Articles, and the document needed to be in MarkDown for github issues, and in html, so naturally the source should be in RakuDoc.

Although numbered headings and items were specified in RakuDoc v1 (aka POD6), they were never implemented. In RakuDoc v2 there was progress, and together with some tweaking of the numitem templates, it was easy to write the RakuDoc source.

However, when we were revising RakuDoc, Damian Conway had some extra ideas about generalising enumeration, including ideas about adding alias definitions so that the numbered block could be referenced later in the text.

Since it had been so easy to tweak the numitem templates, I thought it would be easy upgrade the specification of RakuDoc v2 to get these generalisations. Was I ever so wrong!!!! Ask a genius with decades of language design experience for a nice design and you get an effusion of ideas and extensions that make RakuDoc better than any editor I have ever used, but with a simplicity that makes it easy for a document author to understand.

Another result of the redesign is to make the underlying specification of RakuDoc much clearer, something I will cover later.

During the development process, my daughter mentioned that her friend had just finished a PhD dissertation and was complaining about how much time she needed to spend reformatting the text because the numbering kept getting out of sync. The problem with most editors is that most of the effort goes on perfecting the user-facing interface, while the underlying format is created ad hoc, and enumeration is an addition. Getting the underlying structure right will make subsequent rendering easier.

Just as the design was ending and the renderer passing most tests, I casually mentioned citations would be a good extension. The "standards" for citations number in the thousands! .oO(The Jabberwocky ) But Damian snicker-snacked his vorpal blade, reducing the monstrous tangle to something easier to use. RakuDoc v2 now has a =citation block and Q<> markup to insert quoted citations. I will cover these additions in the next blog (as in: when I get the renderer to work with the new ideas).

Back to enumerating RakuDoc, here are some examples to illustrate the new functionality.

Suppose you have a formula (or code or map or table or item in a list) and you want to number it, and then reference the number in the text? Then another formula gets added into the text before, well you would want the references to update as well.
Suppose you have tables that you want to be enumerated separately from headings? But also by preference you want them to be enumerated in sequence with headings? That is the prefix to the table enumeration is the heading number.
Suppose you want Chinese, Roman, or Bengali numbering?
AND suppose you also want the numbering to be a mix of numerals and other characters, such as brackets? This is a common requirement in legal documents.
Suppose you want Arbitrary words before after or in between the numbers? For example, Article 1., Article 2. etc.
Suppose you want to have the paragraphs numbered? Sometimes you might want the paragraphs numbered in sequence from the start of the document, and sometimes you want the numbering to restart after a new heading.
Suppose you want to have the Tables and the Formulae numbered in sequence? Or perhaps, for a section that explains some aspect of one formula, you want to number the formulae separately, but then return to the original sequence after that section?

The updated RakuDoc v2 allows for all of these possibilities, while also providing for sensible default option values.

Moreover, RakuDoc allows for custom blocks, eg. LeafletMap - a map block that exposes the marvelous Leaflet library. Now it is an automatic part of the specification that prefixing a custom block with num (numLeafletMap) will number the caption of the block.

An HTML rendering of the new RakuDoc v2 specification containing the enumerated functionality can be found at RakuDoc enumeration branch

Understanding the RakuDoc paradigms

In the specification of RakuDoc, there are discussions about directives, blocks, metaoptions, and a section on =config.

Since the syntax for a directive is almost the same as the syntax of a block, it is not immediately apparent what the difference is. As we developed the generic enumeration, it became much clearer what the difference is.

Furthermore, the older specification covered multi-level headings and items. This functionality is now extended to all blocks, so =numtable2 or =numcode3 will be numbered in parts from the previous instance of =table (the equivalent of table1) or =code. This means that we need to carefully distinguish between the block base (eg., table or code), and the block level (eg., 1, 2, 3). Together the base and the level create a blocktype. These distinctions, although implied, were not so clear in the older specification.

Let's return to the =config directive. The first parameter of =config is the name of a blocktype. Note that the num prefix is not a part of the blocktype and only indicates that the enumeration associated with the instance needs to be rendered. Consequently, the presence or absence of 'num' in the config parameter has no significance. The next =config parameters are all, and may only be, metadata options.

The function of the =config directive is to distribute the named metadata options, and their values, to the named blocktype. In addition, the same metadata option can be specified on a blocktype instance (using either the =for or =begin directives), in which case it takes precedence over the option in the config. A consequence of this paradigm is that each metadata option has to have a semantic significance in the context of a blocktype instance.

One of the design aims was to give a document author the choice of restarting the enumeration for some blocktype when another blocktype is encountered. For example, the original specfication for =numitem included the idea that whenever a sequence of =numitems was encountered, they would form an ordered set, and that if another block, such as a paragraph, was encountered, the enumeration would be restarted.

This means that the occurrence of a =head restarts the item counter. This cannot be controlled within the handler of a block. Consequently, any option affecting block counting needs to be distinct from the rendering of the blocks themselves.

After some design iterations, a new directive called =counter was introduced. A directive, as opposed to a block, affects all subsequent blocks in the RakuDoc source, within the same scope. A block, and the metadata options operating on the blocktype, only affects its immediate contents.

Further, it became clear that a block and its counter were different objects, although for simplicity they have the same name. For most needs, an author does not need to know that a counter and a block are different, except that when a new counter is needed, the difference becomes necessary. In a similar way, for many purposes it doesn't matter whether a number is an integer or a string, except when it does. Raku allows for things to become complicated when the author needs it to be.

How to experiment with enumerations?

To make it easier to experiment with RakuDoc and the enumerated functionality, I have developed a Docker image called browser-editor. It is based on an Alpine image and contains raku, Cro, and the latest version of Rakuast::RakuDoc::Render (currently not yet in the fez system).

Using podman, the image can be put into a container and run locally on Linux based systems, thus:

podman pull docker.io/finanalyst/browser-editor:latest
podman run -d -v .:/browser/publication --rm -name rb docker.io/finanalyst/browser-editor:latest

For Mac silicon, a small change is needed to indicate the platform inside the docker image is based on Linux, thus:

podman run -d -v .:/browser/publication --rm --platform linux/amd64 --name rb docker.io/finanalyst/browser-editor:latest

In both cases the container is given the arbitrary name rb, and so when it is time to stop the container, the following is sufficient:

podman stop rb

The directory that the container is started from will then linked to the container's /browser/publication directory, and changes and new RakuDoc source files will be saved in that directory.

A RakuDoc source can then be edited and the HTML rendering is created on the fly, by pointing a browser at (setting the browser URL to) localhost:3000.

You should see something like the following in the browser:

Two frameworks exist: one needing no internet connection - a minimal single file rendering, and another that uses plugins to expose some useful third-party libraries and the more sophisticated Bulma CSS framework. The choice is toggled using the 'online' button.

Try selecting 'online' and copying in the following test source by Damian Conway to see some different effects.

=begin rakudoc
=TITLE Taster source
=numitem One
=numitem Two
=counter item :restart(42)
=numitem Three
=counter item :restart
=numitem Four
=counter item :prefix<head3>
=numitem Five
=counter item :!restart
=para ???
=numitem Six

=numpara
S<Now is the winter of our "no content"
Made glorious summer by Camelia’s bloom;
And all backlog that lour’d upon our docs
In deep commits is buried, link’d, and tagged.
Our nightly builds now wear triumphant green,
Our failing tests to passing smiles are turn’d;
Grim sighs of “ere next Yule” have chang’d to grins,
And hacker dread to blog posts boldly strung.>

=numpara
S<But I—long nurs’d on RFCs and hope,
Deform’d by specs that shifted as I read,
Unfit for idle scripts or stable sleep—
Am set, since long delays are now no more,
To ship Raku...and break the world anew.>

=numcode
$x = any <1 2 3>;

=for numcode :caption<Same thing, just labelled>
$x = any <1 2 3>;


=for numformula :caption<This means nothing!>
x^2 = y_j + \sum z_i

=for numformula :caption<This means nothing!> :alt< xH<2> = yJ<1> + E<GREEK CAPITAL LETTER SIGMA> zJ<i> >
x^2 = y_j + \sum z_i

=for numformula :alt< (1+x)H<n> = E<GREEK CAPITAL LETTER SIGMA> H<n>CJ<i> xH<i> > :caption<This is actually right>
(1+x)^n = \sum_{i=0}^n {n \choose i} x^i

=for numinput
Hey type something:

=numoutput
You typed: hsdkhldkhskdhasd

=counter numtable :restart(33456)
=begin numtable :caption<Encryption table> :form<Example %R: %C:pc>
=row
=cell A
=cell B
=cell C
=row
=cell D
=cell E
=cell F
=end numtable

=end rakudoc

This should look like the following (or at least the top part).

The example shows some features like numbering paragraphs, code examples and tables. But you will also see the difference between the two rendering contexts. The online version has access to an online latex rendering resource, so the formula are nicely rendered, whilst the off-line version only renders to the raw formula or a character-based alternative - if provided.

Coding the examples in the introduction

Adding an alias to a block

Suppose we want to enumerate a code sample and then refer to it later, we can do the following:

=begin rakudoc :!toc
=TITLE Enumerations
=for numcode :numalias<SAY_EX> :lang<raku>
my %h = <one two three> Z=> ^3;
say %h;

=numoutput {one => 0, three => 2, two => 1}

=for numcode :numalias<PUT_EX> :lang<raku>
my %h = <one two three> Z=> ^3;
put %h;

=for numoutput                                                                                                                                    
one     0
three   2
two     1

The difference between A<SAY_EX> and A<PUT_EX> is that C<say> and C<put> use different methods to convert the C<%h> structure into printable strings.
=end rakudoc

In the browser-editor image, we will get an HTML rendering something like:

The =begin rakudoc :!toc and =end rakudoc are used to top and tail the RakuDoc example above because the HTML renderer expects a complete Raku program, and a RakuDoc source (currently) needs the RakuDoc to be within a =rakudoc block. The :!toc switches off the automatic Table of Contents that the HTML utility of the Rakuast::RakuDoc::Render distribution generates. (Try removing :!toc)

Prefixing an enumeration with another counter

Suppose we want out tables to have enumerations that align with the enumerations of the headings. So this means we are requiring a different sort of behaviour from the counter associated with the table block base. Consequently, we need to use the =counter directive. Also note that we are prefixing with the head2 block, which has an implicit prefix of head1.

=begin rakudoc :!toc
=TITLE Prefixes
=counter table :prefix<head>
=numhead First title
=numhead2 Sub first title
=for numtable :caption<First table>
| one | two |
=for numtable :caption<Second table>
| three | four |
=numhead Second Title
=counter table :restart(6)
=for numtable :caption<Third table>
| five | six | seven | eight |
=for numtable2 :caption<Subordinate table>
| nine | ten |
| nine | ten |
=end rakudoc

Comments:

A table must have some content, and the 'visual' table format is the minimum possible
In order to make the prefix numbering a bit clearer, I manually restarted the table counter to 6 before Third table.

Enumerations with different numbering systems

To illustrate the multi-lingual ability of RakuDoc (and Raku in general), lets use Chinese, Roman and Bengali for multilevel headings.

=begin rakudoc :!toc
=TITLE Some non-Arabic numerals
=config head :form< %Z %D >
=config head2 :form< ｢%Z｣%R %D >
=config head3 :form< ｢%Z｣%R｢%B｣ %D >
=numhead First title
=numhead Second title
=numhead3 Sub-sub-head one (implies the sub-head)
=numhead3 Sub-sub-head two
=numhead2 Sub-head first explicit
=numhead2 Sub-head second explicit
=for numformula2 :form<%T %Z.%R. >
\begin{align*}
\sum_{i=1}^{k+1} i^{3}
&= \biggl(\sum_{i=1}^{n} i^{3}\biggr) +  i^3\\
&= \frac{k^{2}(k+1)^{2}}{4} + (k+1)^3 \\
\end{align*}
=end rakudoc

This will produce an HTML rendering a bit like

Including arbitrary text

The :numform option is very flexible and in fact the brackets in the previous example are arbitrary text. The following RakuDoc source

=begin rakudoc :!toc
=TITLE Arbitrary text in enumeration
=config head :form< Article %N. %D >
=config head2 :form< Article %N.%r. %D >
=config head3 :form< Article %N.%r｢%Z｣ %D >
=numhead First title
=numhead Second title
=numhead3 Sub-sub-head one (implies the sub-head)
=numhead3 Sub-sub-head two
=numhead2 Sub-head first explicit
=numhead2 Sub-head second explicit
=end rakudoc

will produce something like:

Numbering paragraphs

In RakuDoc, paragraphs as in all text editors, are strings of text ending in a blank line or a new block (for RakuDoc). There is no need to mark a paragraph block, but it will be equivalent to a block marked =para.

By default, each paragraph is numbered from the start of the document. In order to see the enumeration, the paragraph does need to be started with a =numpara.

For this example, we want to number paragraphs in each section marked with a heading. So we need to tell the counter which conditions it is restarted by; in this case after every =head (by default a bare block name has a level of 1). There are both :restart-after and :restart-except-after conditions, but that's all we'll say about them here.

The RakuDoc source is

=begin rakudoc :!toc
=TITLE Restarting after headings
=counter para :restart-after<head>
=head First heading
=numpara First line
=numpara Second line
=numpara Third line
=head Second heading
=numpara First line 2
=numpara Second line 2
=head2 Subordinate heading does not restart the paragraphs
=numpara Third line 2
=numpara Fourth para
=end rakudoc

The source will yield something like:

Custom counters and block scopes

We want to number code and formulae using the same counter, but also we want to explain some formula with a special numbering that is then forgotten.

Since all =config and =counter statements are scoped, all we need to do to get the special numbering is to create a =section, which introduces a new block scope. The previous config/counter options continue, unless explicitly overridden.

However, it would not be useful if upon exiting a section, the numbering of a counter reverts to the value before the section. So there is a subtle distinction between the counter and the value of the count it contains. The counter options, such as its prefix, and when it is restarted, are scoped, but the value of the counter is not scoped. In order to reset the counter value, the counter has to be reset.

RakuDoc v2 clarified the idea of custom blocks. Previously it was implied that developers could have their own blocks, but in V.2 the difference between built in and custom blocks was clarified: custom blocks must contain a mix of upper and lower case letters (more precisely characters with the Unicode properties Lu and Ll).

The enumeration options use this idea by creating custom counters. Since this is something that managed by a block, the :counter option is provided to the appropriate blocks using a =config directive.

=begin rakudoc :!toc
=TITLE Using custom counnters
=config formula :counter< FormulaeTables >
=config table :counter< FormulaeTables >
=numhead First title
=numtable 
| one | two | three |
=numtable
| more | tabula data |
| 5 | 6 |
| 7 | 8 |
=numformula E = mc^2
=numformula e^{\pi i} = -1

=begin section
=config table :counter< Derivation > :form<Aside: %T %N>

=numhead2 Some explanation
=numtable
| this | data | is | more | detailed |
| 1 | 2 | 3 | 4 | 5 |
=end section

=numhead Continuing
=numtable
| here | we | resume |
=end rakudoc

This will produce something like

Afterword

We have only just completed the revised specification, and the Renderer has only just been developed. It is inevitable that there will be flaws, and the styling choices may not be liked by everyone.

Even so, we think this new upgrade of RakuDoc v2 will prove to have a much wider applicability than just a documentation aid to Raku programs.

The docker image allows anyone to experiment with the new RakuDoc functionality immediately without needing to install the whole Raku / Cro / Rakuast::RakuDoc::Render stack and some dependencies, such as Dart Sass, locally.

Create a minimal site with Elucid8

Richard Hainsworth — Tue, 02 Sep 2025 21:59:43 +0000

The Elucid8 system can be used to create websites based on RakuDoc. The article is to show how to create a minimal website.

Elucid8 ("elucidate") is still being developed, and more information can be found in the Github elucid8-org repositories.

The following need to be present:

A recent version of Rakudo v.2025.01 or later.
Dart sass - see Sass website for installation instructions. (It will be used by a Elucid8 plugin, so it should be globally visible)
Elucid8::Build - zef install Elucid8::Build
Elucid8::Run-locally - zef install Elucid8::Run-locally

Assuming that the zef installs bin/ files to a location in the PATH, then the utilities in the next section should run without problem.

Setting up minimal site

In an empty directory (to be concrete, lets call it webdir), which will be the root for the website build, run the following

elucid8-setup
gather-sources
elucid8-build
run-locally

Then point a browser at localhost:5000

That is all for the minimum site.

Now change the file site-source/en/index.rakudoc.

Then run elucid8-build; run-locally again to see the changes.

Explanation of steps

Step 1 Minimal files

Here elucid8-setup copies resources in the Elucid8::Build distribution to create the minimal configuration entries in webdir,
a sample text and some minimal plugins.

The directory structure under webdir will be something like

- config/
  - 01-base.raku
  - 02-plugins.raku
  - 03-plugin-options.raku
  - 04-repos
- misc/
- site-sources/
  - en/
    - index.rakudoc
    - examples.rakudoc

Since Elucid8 is being built from the bottom up to be multi-lingual, it is intended that all of these directories can be named in a local variation. The file config/01-base.raku contains the tokens that are used within Elucid8::Build.

Step 2. Gathering sources

It is intended that each set of language sources will be in a different repo. gather-sources looks at config/04-repos for files and repo information, clones repositories, runs git blame against them and stores a file description in misc/.

After this step, website will contain the file misc/repo-info.rakuon.

For the minimal website, the RakuDoc v2 specification is pulled from the Raku repository. This document also shows many of the features of RakuDoc.

The config/04-repos shows an example of how to map documents from the repository to subdirectories within publication.

Step 3 Build

At this step, the build process starts. A processor engine is created that uses the plugins described in config/02-plugins. These are run with options described in config/03-plugin-options.

After this step, misc/ also contains ui-dictionary.rakuon, which will have the English version of the UI tokens. When this dictionary is appropriately edited, other languages will be available.

Elucid8 makes a distinction between the language of the UI and the language of the contents.

A new directory publication/ has now appeared, and this will contain the HTML version of the index file.

Step 4

At this step, a Cro app is run that takes the HTML in publication and serves the files to localhost:3000.

Customisation

There are many ways to customise the site:

make sure to change config/03-plugin-options and the root-domain field of SiteMap, so that the SEO map points to your website.
adding RakuDoc sources, remember to add links in index.rakudoc
adding plugins to create new RakuDoc blocks
use the ListFiles block to automatically include in index.rakudoc all the other RakuDoc sources in your local-sources/ directory.

Examples of what can be done can be seen in Sandpit repo, for Raku documentation

A coordinated dance to identify the editor

Richard Hainsworth — Fri, 25 Apr 2025 14:47:44 +0000

Adding login logic - rationale

In a previous article, I described how to take a web page source file, edit it, then send the edited source back to Github.

In that post, the authorisation token id-token was one I generated for myself as a Github user. It would also be possible to assign to the hallowed glade (see previous article) an authorisation for a pseudo user or bot specifically set up to take user generated editing.

However, when the edit is merged into the source, the author of the commit would be the bot, and not the author of the edit.

Suppose, though, we want a 'credits' page listing authors and the number of commits they have made. We could run a command in the repo such as:

git log --pretty="%an %n" | sort | uniq -c | sort -n -r

and filter the results into a table. But authors who have entered our hallowed glade will not be listed.

Another concern is spamming. If we require a community user to have a Github identity, then Github will handle authorisation. If there is spamming by a forest troll, it is in Github's best interest to hold that troll to account.

This post explains (mostly to myself) the steps that are needed to insert a login layer.

When I finally got the whole thing working, it was like a weird dance between three different entities. Looking at the documentation for the first time, all the steps seemed odd and unrelated, but once they were all put into motion, there is a logic to the whole thing.

Overview

We already have a hallowed glade, which is the suggestion_box server. The server runs in a docker container, and has a Cro server to handle input from a websocket, a Cro client to interact with Github to create a new branch, store the edited file, and raise a Pull Request.

Github allows the owner of a repo to register a 'Github app' and assign it permissions. So, the token granted during the authorisation process is a combination of the permissions of the app and those of a user.

It is always scary for a user to have to authorise someone else to do something on their behalf, so here is what Github says about the authorisation token:

A token has the same capabilities to access resources and perform actions on those resources that the owner of the token has, and is further limited by any scopes or permissions granted to the token. A token cannot grant additional access capabilities to a user. Github authorisation documentation

Coordination

Here is diagram of the interaction between the browser, the server (in our case suggestion_box) and Github from a great article by Tony on OAuth2:

Suppose we add in the request that if an editor wants to make edits on several documents (which means the page is refreshed), then it is inefficient for Github to generate an id-token for each page.

We can keep an object in local storage with the name of the editor and the time the first submission is accepted.

Note that Github issues its web tokens for a fixed period of time, by default 8 hours.

A pre-requisite is that the Github app, which is our suggestion-box is already registered with Github.

Steps

The brief laid out above suggests the following series of steps:

When a submission is made, the editor field in the submission form must be consistent with Github rules:
- Github has a user name policy of only alphanumeric characters + -, with a minimum of three characters and a maximum of 39.
- Letters are case insensitive and other characters are replaced by -.
The browser checks to see if the editor has made a successful submission within the last 8 hours, and that 10 minutes is still left.
- if there is no or insufficient time, the local storage is deleted.
If the editor has not been verified, the editor is sent to the Github login page, together with a state string that contains information for the suggestion_box to be able to issue a successful submission message back to the browser.
Whether or not the editor has been verified, and in parallel with the Github verification, the suggestion information is sent to the suggestion_box as outlined in my previous article.

Inside the suggestion_box server:

The editor is verified to see whether an id-token with enough time (10 minutes) is available. If so, the suggestion is queued, and a successful message is sent back together with the remaining time on the token.
If there is not enough time left, a failure message is sent back from the server to the browser.
If there is no id-token and ten minutes have passed after the suggestion was received with no message from Github, a failure message is sent back to the browser.
When a message is received from Github, it is compared with waiting suggestions. If there is not a match between the state field of a suggestion, the Github message is ignored.
If the state matches a waiting suggestion, the suggestion-box server starts the process to get a id-token. If successful, the suggestion is queued and a message is sent back to the browser containing the success and the expiration time.
- if unsuccessful, a failure message is sent back to the browser

In this scheme,

the browser never sees the editor's id-token.
the actual name provided by the editor in the suggestion form does not have to be the Github name of the editor because the Github authorisation is independent of the editor's name, but future submissions must use the same editor name to use the id-token, which is time-limited in any case.
an editor may edit several files in one session.

The Cro setup

In the previous post, the Cro app had a route for the websocket and a Client section to handle putting suggestions into Github.

We need to modify the Client section to use the editor's id-token, but essentially it remains the same.

We also need to add a route which is used by Github to send login information to the server. In addition, there needs to be a way for the webserver route and the authorisation route to interact.

These modifications imply a shared resource to match editor-name, id-token, id-token expiration date. Since Cro assumes concurrency, the storage has to be thread-safe and ensure that only one thread at a time can access it.

Although the Cro documentation uses OO::Monitor for this purpose, I prefer the simpler Method::Protected module. The is protected trait ensures that only one thread at a time can access the shared resource. For example,

use Method::Protected;
class Editor-Store {
    #| has key = editor, with two attributes :token and :time (a Date::Time)
    has %!storage;
    #| if False, makes sure editor key is deleted
    method is-editor-active( $editor --> Bool) is protected {
        return False unless %!storage{ $editor }:exists;
        return True if %!storage{ $editor }<expiration> > (now.DateTime + Duration.new(10 * 60));
        sink %!storage{$editor}:delete;
        return False
    }
    #| if there is no editor, an emptry string is returned
    method get-token( $editor --> Str) is protected {
        if self.is-editor-active( $editor ) { %!storage{$editor}<token> }
        else { '' }
    }
    #| expiration date
    method expiration( $editor ) is protected { %!storage{$editor} }
    #| returns the remaining time in seconds as an integer
    method time-remaining( $editor --> Int ) is protected {
        if self.is-editor-active( $editor ) { ( %!storage{ $editor }<expiration> - now.DateTime).Int }
        else { 0 }
    }
    #| adds editor
    method add-editor( Str $editor, DateTime $expiration, Str $token ) is protected {
        %!storage{$editor} = :$expiration, :$token;
    }
}

The Websocket route of the Cro app needed refactoring completely. Instead of getting all the information for an edit from the browser, typically, the edit suggestion comes first but the editor and their id-token comes later. So the Webocket needs to create a promise and also to put a timer on it.

My first-draft solution is (the code needs to be cleaned up somewhat):

get -> 'suggestion_box' {
            web-socket :json, -> $incoming {
                supply whenever $incoming -> $message {
                    my $json = await $message.body;
                    # first filter out the handshake signal for opening a websocket
                    if $json<loaded> {
                        say strftime(DateTime.now, '%v %R') ~ ': connection made'
                            if $debug;
                        emit({ :connection<Confirmed> })
                    }
                    else {
                        if $debug {
                            say strftime(DateTime.now, '%v %R') ~ ': got suggestion, now at ' ~ +@suggestions;
                            for $json.kv -> $k, $v {
                                say "KEY $k =>\n$v"
                            }
                            say "edit suggestion finished\n";
                        }
                        my $response = sanitise( $json ); # sanitise returns an error message or 'ok'
                        my $editor := $json<editor>;
                        if $response ne 'OK' {
                            emit( {
                                :timestamp( DateTime.now.Str ),
                                :$response,
                                :$editor,
                            })
                        }
                        # handle the socket with an active editor
                        elsif $store.is-editor-active($editor) {
                            say strftime(DateTime.now, '%v %R') ~ ': editor is registered' if $debug;
                            # too little time left for token
                            if $store.time-remaining( $editor ) <= 10 * 60 {
                                say strftime(DateTime.now, '%v %R') ~ ': not enough time' if $debug;
                                emit( {
                                    :timestamp( DateTime.now.Str ),
                                    :response<TooLittleTimeOnToken>,
                                    :$editor,
                                })
                            }
                            else {
                                say strftime(DateTime.now, '%v %R') ~ ': handling with stored token' if $debug;
                                $json<id-token> = $store.get-token($editor);
                                @suggestions.push: $json;
                                emit( {
                                    :timestamp( DateTime.now.Str),
                                    :response<OK>,
                                    :$editor,
                                    :expiration( strftime($store.expiration($editor), '%v %R'))
                                } )
                            }
                        }
                        # the editor does not have a token, but may have in some period of time
                        else {
                            say strftime(DateTime.now, '%v %R') ~ ': editor without authorisation' if $debug;
                            my $timestamp;
                            my $response = 'NoAuthorisation';
                            my $expiration = '';
                            my $token = '';
                            my Promise $tapped-out .= new;
                            my $tap = $see-new-auths.tap( -> %a {
                                if %a<editor> eq $editor {
                                    $timestamp = DateTime.now.Str;
                                    $response = 'OK';
                                    $token = %a<token>;
                                    $expiration = %a<expiration>;
                                    $tapped-out.keep;
                                }
                            });
                            await Promise.anyof(
                                $tapped-out,
                                my $timer = Promise.in($auth-wait-time).then: {
                                    $response = 'NoAuthorisation';
                                }
                            ).then( { $tap.close } );
                            if $response eq 'OK' {
                                $json<id-token> = $token;
                                @suggestions.push: $json;
                                say strftime(DateTime.now, '%v %R') ~ ": authorising suggestion {$json.raku}, now at " ~ +@suggestions if $debug;
                            }
                            emit( %( :$timestamp, :$response, :$expiration, :$editor) )
                        }
                    }
                }
            }

The $tapped-out Promise is created so that it can be kept with .keep inside the code that listens for an authorisation event.

Then a separate Promise composed of the $tapped-out Promise and a timer Promise is created so that whichever comes first triggers the next step. If the timer exits before an authorisation, then the edit suggestion is discarded, otherwise it is combined with the id-token and queued.

In the code section above, the webserver is a supply and when the emit sub is called, the hash argument is mapped by Cro into a JSON object and returned to the browser that has connected to the Cro app. So it can be picked by the Javascript's websocket's onmessage function, and the data used by the browser program.

Authorisation event

When a Github app is registered, a route is supplied for the authorisation data. Consequently, the server section of the Cro app has to be set up to service this route. I chose the route /raku-auth (and in my code comes before the websocket, but since we are dealing with concurrent processes, the order of websocket and raku-auth is irrelevant):

get -> 'raku-auth', :%params {
            CATCH {
                default {
                    content 'text/html', '<h1>Raku documentation</h1><p>Authorisation error.</p><p>Please report</p>';
                    say 'error is: ', .message;
                    for .backtrace.reverse {
                        next if .file.starts-with('SETTING::');
                        next unless .subname;
                        say "  in block { .subname } at { .file } line { .line }";
                        last if .file.starts-with('NQP::')
                    }
                }
            }
            my %decoded = from-json( base64-decode( %params<state>).decode );
            say strftime(DateTime.now, '%v %R') ~ ': got from Github params: ', %params , ' state decoded: ', %decoded
                if $debug;
            my $editor = %decoded<editor>;
            my $resp = await Cro::HTTP::Client.post(
                "https://github.com/login/oauth/access_token",
                query => %(
                    :$client_id,
                    :$client_secret,
                    :code( %params<code> ),
                ),
            );
            # Github returns an object with keys access_token, expires_in (& others not needed)
            my $body = await $resp.body;
            my %data = $body.decode.split('&').map(|*.split("=",2));
            # first store the data for future suggestions
            my $token = %data<access_token>;
            my $expiration = now.DateTime + Duration.new( %data<expires_in>);
            $store.add-editor($editor, $expiration, $token );
            # next put the data in a stream for suggestions that have already arrived
            $auth-stream.emit( %( :$editor, :$expiration, :$token ) );
            content 'text/html', '<h1>Raku documentation</h1><p>Editing has been authorised.</p><p>Thank you</p>';
        }
        get -> 'suggestion_box' {

The CATCH phaser is only ever triggered if there is an error processing the route. If it is triggered, then the string after content is sent back to Github, which displays it in the tab containing the authorisation button. The HTML could be improved.

At the end of the code, the content sub similarly sends back an HTML string to indicated authorisation is successful.

When a route has a query appended to it, the Cro route function get captures the data into a named hash, which I have called %params. Github recommends that a state variable is sent when sending the user to Github for authorisation. I have chosen to send the name submitted by the editor with Base64 encoding. Since the editor name in the form and the user's Github id could be different, this adds some complexity to improve security.

I have to say that the Cro syntax is easy to work with. When the data is transmitted as a JSON, there is the named :body field, and with the data is transmitted as a query, there is the named :query field. Otherwise the get sub has a consistent syntax. Compare this to a cURL command.

The parameters sent to the route include a code field, which is then returned (again as a query) to Github. When it receives the code, it returns the id-token for the editor. This two-fold handshake makes it difficult to impersonate someone else.

In addition, Github returns the number of seconds the id-token is valid for. So this needs to be combined into a DateTime both for the suggestion-box server and the browser.

Finally, the editor name, id-token and expiration date are both stored in a thread-safe Hash, and placed in an event stream into which the websocket has tapped.

Raku's concurent structures make it easy to set up the event loop into which the raku-auth route injects information and the websocket listens for information. At the start of the code we have

my $auth-stream = Supplier.new;
my $see-new-auths = $auth-stream.Supply;

As can be seen from the code fragments, the raku-auth route has the line $auth-stream.emit( %(...) ) and the websocket code has the line my $tap = $see-new-auths.tap( -> %a { ... }).

The first stanza supplies an item, in this case as hash, while the second listens for all items supplied. The second then determines which item to react to.

Keeping secrets

One of the pieces of data required by a Github app is a 'secret' for the app. There are also several items to be provided to the Cro app.

Since this Cro app is intended for a docker container, the configuration data is conveyed in Environment variables. So at the start of the app, several secret variables are extracted as (just an example here):

my $client_id = %*ENV<CLIENT_ID>;

and these data can be temporarily saved in an environment file env-file. Then when the docker image is invoked the data can be supplied:

sudo docker run -d --rm --env-file env-file my-docker-image

Treading on toes

As can be seen from the diagram above, getting authorisation is delicate dance, and it took me days to stop the browser, Github and the suggestion-box treading on each others' toes.

First, the interaction between Github and the suggestion-box server to exchange a code for an id-token are all conducted using the query format. That is a url ending ?data-item=stuff&item-two=nonsense, which is in turn Base64 encoded. All the Github API calls use JSON data with authorisation headers.

In hindsight, we can recognise that the OAuth protocol is an industry standard, while the Github API protocols are not, so can be different. But it took me a while to work out what the strange data was being received by the server. I did not come across this behaviour as being explicitly documented.

Second, Github requires that the suggestion-box server is registered by the owner of the repo as a Github app and specific permissions need to be allocated to it. There are dozens of permissions in over a dozen categories and the correct ones have to be given to the Github app. I thought - mistakenly - that the permissions for Pull requests was what I needed.

Actually this was the last bug I had to overcome and it took a couple of days to figure out that the error was not in the server code, but that I had not allocated enough permissions.

To summarise, the suggestion-box server uses four separate Github API calls and the permissions are different:

api.github.com/repos/{$repo-name}/git/ref/heads/main to get the commit sha for the repo-name - this is public data and the permission is mandatory for an App, so it does not need to be specifically added
api.github.com/repos/{$repo-name}/git/refs (with data) to create a reference - this requires the Contents permission with read/write
api.github.com/repos/{$repo-name}/contents/{$file-path} to supply the edited - this requires the Contents permission
api.github.com/repos/{$repo-name}/pulls to create a pull request of the new branch - this requires the Pull Request permission

Final thoughts

Some 'simple' requests have complex solutions. Although Cro and Raku's concurrent structures take a while to understand, they are easy to apply.

Editing RakuDoc, CRO'ing it to Github

Richard Hainsworth — Mon, 14 Apr 2025 20:54:26 +0000

All I want is to correct a one-letter typo

The ask, expressed by our website users, is simple enough, edit a source from a GitHub repo in a browser, then save the edited version back.

How difficult is that? It's easy to point the browser at the file in the repo, and GitHub has an editor ...

The problem comes from authorisation, and bad actors wanting to add stupid stuff to files - like: I'm the brilliantish shop for high-ent wazzoo's. Come take a look at http://rip-me-off.darkweb.cosmo

Github's editor is easy to use - if you have commit permissions on the repo, otherwise:

you need to clone the repo,
correct the one letter typo in a single file,
raise a PR. Which is fine if you know what a 'clone' is, or a 'repo' is, or a 'PR'.

But its only one letter!??

Also, our website text source is written in RakuDoc (not MarkDown, but that's another story) and so it would be nice to be able to see what the change looks like. As always a seemingly trivial mistake, such as deleting a < in the wrong place, can have severe effects.

So, it would be nice to see an HTML rendering of the edited text before submitting it back.

The first steps into the forest

It seems doable:

Get the content of the file from Github - there is a simple URL and the Javascript fetch function.
Put the content into an online editor - there are a few editor libraries, which give you all sorts of control; I chose one.
Get the content back from the editor at some point, send it to a renderer - I put my Rakuast::RakuDoc::Render into a docker container, wrote a very simple Cro app that has all the logic for a websocket. The app expects RakuDoc source, and sends back the rendered HTML, which is then added into a div next to the online editor.
Send the edited version of the file back to Github - oh. I'm lost in the forest.

I'll not explain tasks 1-3 above and focus on task 4.

The path through the forest

While getting the content is easy, returning it is hard. The API documentation is written for developers who understand Github and HTTP requests.

I spent two days wandering though the API documentation, searching for tutorials, (and Github's own discussion forum seems to be flooded with trolls, so is not a good place to look).

I even resorted - gasp - to ChatGPT. Whilst the code it suggested was not remotely what I wanted, the 'solutions' demonstrated a remarkably simple truth. Perhaps I should have realised it on my own, but in all my documentation searches I had not read a hint about it.

The simple truth: getting a patch of a file to the repo maintainers could only be done in a sequence of steps, not the single step that I was looking for.

It is obvious that Github wants to ensure that only authorised users can change the state of a repo. Authorisation is accomplished by accompanying any request for a change with a token that has a limited duration, is unique to a recognised user on Github, and has permissions for the repo.

While it is possible to get and use such a token for an arbitrary user, by sending them to Github to log in, the process is an added layer of complexity. For simplicity, we shall consider that a token has been generated and call it id-token. (For development, I used one generated for myself)

The steps on the map

In generalised terms, here are the steps to sending a suggestion for editing a file to the Github repo. (I'll explain some of the jargon with each step).

Obtain the contents of the file and its sha from repo.
- Github stores all the information developers use in files. Each is tagged with a sha, and is located in a repository or repo.
- Actually Github has a two layer structure of owner/repo-name, so in this article, when I say repo, I mean the combination of owner/repo-name. For example, the site I am working on uses the source from the Raku/doc Github storage, where Raku is the organisation that owns the storage, and doc is the repo in which Raku holds its documentation suite.
- The sha is the encrypted sum of the entire file, and is presented as a 40 digit hexadecimal string. The maths of shas is quite interesting but not relevant here. Suffice it to say that if a file is changed at all, the shas of the two files (before the edit and after it) will be different (for the pedantic, I'll add 'with high probability').
- In fact, this step can all be done in the browser because there is no change in state of the Github storage.
Obtain the latest sha or commit for the repo.
- Just as the contents of one file at one point in time can be identified by a sha of the file, so too can the contents of the whole repo.
Create a new branch or reference point in the repo.
- In order to retain both the existing content of a repo and the suggested new content, the new content is held in a new part of the repo, and this new 'section' is called a branch. It can also be thought of as a reference point in the history of the repo. At some point the maintainer of the repo can accept the new content (merging it into the main part of the repo), or reject it.
- Creating a branch changes the state of the repo, so Github only allows a recognised agent to do this. So, this step has to be accompanied by an id-token.
- The information needed by Github for this step is
  1. The name of the repo
  2. The sha of the repo
  3. An id-token
  4. A name for the branch
  5. A description of the branch
Move a copy in BASE64 format of the edited file to the new branch
- BASE64 is a way of coding a file so that it can be transmitted safely across the internet.
- This step needs
  1. The name of the repo
  2. The name of the branch
  3. The path of the file inside the branch
  4. The sha of the original file
  5. The new contents of the file in BASE64 format
  6. An id-token
Raise a PR for the branch
- A PR is a pull request, and it is a suggestion to the repo maintainer to merge the suggested changes of the file into the main content of the repo.
- This is the end result we want.
- A branch may contain changes to many files (not just one), and so a PR is for the whole branch (for the pedantic: 'typically').
- This step needs
  1. An id-token
  2. The repo name
  3. The branch name
  4. A title for the PR
  5. A description of the PR

Even though only one letter in one file may need to be changed, the API is set up (reasonably) for many changes to many files.

Beware ... the trolls and bogey men

Can we do this in the browser? Uh, well ... yes, but No.

The problem is that everything in a browser can be examined by the world. So if you put an id-token in a browser, it can be extracted from the browser, and then a bad actor can use the token to do silly stuff.

So, each of the steps above which require an id-token have to be executed from a place that can be defended, such as inside a container where the id-token can be hidden and renewed on a regular basis.

Creating a hallowed glade in the forest

The solution is to create a safe location - hallowed glade - in which the id-token can be stored, and from which the calls to Github can safely be executed.

So let us create a container with a Cro application. The container will have a websocket to accept the edited file from the browser, and then separately interact with Github.

The Cro documentation was written by a genius for developers with experience, unlike me. They are daunting and confusing - it took days of work to figure things out. However, Cro does make it easy to set up the steps listed above.

The first thing to realise is that Cro::HTTP::Client and Cro::HTTP::Server / Cro::HTTP::Router are - at least externally - independent of each other, but we will need both.

We will need the Server to listen for the data coming down the websocket, and then Client to move the edited file to the Github repo.

Since step 1 above is done in-browser, it needs to be implemented in Javascript. I will not deal with that here, except to say, that when the content is fetched from Github, the sha of the file is saved. The in-browser code finally sends along a websocket a JSON object with the following fields:

repo - the owner/repo-name combination
path - the path of the file inside the repo
sha - the sha of the file
content - the edited content in BASE64 form
editor - the person editing the file
comment - why the edits are made
patch - a Patch between the original and edited files

The last three are not strictly needed, but can be used for some sanity testing.

Lets assume the web socket has a route suggestion_box, then the following is the code for a Cro application to get the data

use Cro::HTTP::Log::File;
use Cro::HTTP::Server;
use Cro::HTTP::Router;
use Cro::HTTP::Client;
use Cro::HTTP::Router::WebSocket;
use DateTime::strftime; # a helper to provide nice time formats

my $patch-limit = 5 * 2 ** 10; # 5k limit of chars
my $comment-limit = 1 * 2 ** 10; # 1k limit of chars
my $pr-update-duration = 10 * 60 * 60; # every ten minutes

my $host = '0.0.0.0';
my $port = 60005;
my @suggestions;
my $busy = False;

my Cro::Service $http = Cro::HTTP::Server.new(
    http => <1.1>,
    :$host,
    :$port,
    application => routes(),
    after => [
        Cro::HTTP::Log::File.new(logs => $*OUT, errors => $*ERR)
    ]
    );
say strftime(DateTime.now, '%v %R') ~ ': starting up.';

$http.start; # this starts the server part of the application

# the following is an asynchronous structure which listens for events
# then does the block for that event. The main event is to stop 
# the app when a control signal is raised.
react {
    whenever signal(SIGINT) {
        $http.stop;
        say strftime(DateTime.now, '%v %R') ~ ': closing down.';
        done;
    }
    whenever Supply.interval($pr-update-duration) {
        # every pr-update-duration seconds, this block is run
        # it calls the code for raising a PR
        unless $busy {
            $busy = True;
            raise-pr( @suggestions.pop ) while @suggestions;
            $busy = False;
        }
    }
}
# these routes are what define the application
sub routes() {
    route {
        get -> 'suggestion_box' { # the route for the websocket
            web-socket :json, -> $incoming {
                supply whenever $incoming -> $message {
                    my %json = await $message.body;
                    if %json<sha> { # no sha, no play
                        my $response = sanitise( $json );
                            # place limitations on the incoming data
                            # add 'cancel' to ignore data
                        @suggestions.push: $json.hash.clone
                                unless $json<cancel>;
                            # store the data to be sent as PR
                        emit({
                            :timestamp( strftime(DateTime.now, '%v %R')),
                            :$response
                        })
                            # send back to the websocket JSON with
                            # a time stamp and a reponse code
                    }
                    elsif $json<loaded> {
                        # send back a handshake when websocket opens
                        emit({ :connection<Confirmed> })
                    }
                }
            }
        }
    }
}
sub sanitise( %edit --> Str ) {
    my $response = 'OK';
    for %edit.keys {
        when 'editor' { 
            %edit<editor> .= subst(/\W/, '_', :g)
                          .substr( ^21 ) 
        }
        when 'patch' {
            if %edit<patch>.chars > $patch-limit {
                %edit<cancel> = True;
                $response = 'TooManyChanges'
            }
        }
        when 'comment' { %edit<comment> .= substr(^$comment-limit) }
        when <sha repo content>.any  {}
        default       { %edit{ $_ }:delete } 
                   # remove unwanted fields
    }
    $response
}
sub raise-pr( %an-edit ) { ... }

The Server code handles all the interaction between the browser and the 'Hallowed Circle'.

Now we execute the remaining steps ( 2 - 5 ) of the Github sequence.

sub raise-pr( %edit ) {
    #| Define the secret token. When the container is run,
    #| the token can be passed as an environment parameter. 
    #| Later, we can refactor to obtain it to identify the
    #| user making the edits.
    my $id-token = %*ENV<ID_TOKEN>;
    my $base = 'https://api.github.com/repos';
    # create a descriptive branch name
    my $branch = strftime(DateTime.now, '%v')
            ~ "_{ %edit<editor> }";
    # get the repo information
    my $resp = await Cro::HTTP::Client.get(
        "/{%edit<repo>}/git/ref/heads/main"
    );
    my %data = await $resp.body;
    my $commit = %data<object><sha>;
    # create a new branch
    $resp = await Cro::HTTP::Client.post(
        "$base/{%edit<repo>}/git/refs",
        content-type => 'application/vnd.github+json',
        auth => { bearer => $id-token },
        headers => [ X-GitHub-Api-Version => '2022-11-28' ],
        body => %(:sha($commit), :ref("refs/heads/$branch"))
    );
    %data = await $resp.body;
    # update file into new branch
    $resp = await Cro::HTTP::Client.put(
        "$base/{%edit<repo>}/contents/{%edit<path>}",
        content-type => 'application/vnd.github+json',
        auth => { bearer => $id-token },
        headers => [ X-GitHub-Api-Version => '2022-11-28' ],
        body => %(
            :sha(%edit<sha>),
            :$branch,
            :content(%edit<content>),
            :message(%edit<comment>),
        )
    );
    %data = await $resp.body;
    # should check to make sure OK
    # Raise a PR for the new branch
    $resp = await Cro::HTTP::Client.post(
        "$base/{%edit<repo>}/pulls",
        content-type => 'application/vnd.github+json',
        auth => { bearer => $id-token },
        headers => [ X-GitHub-Api-Version => '2022-11-28' ],
        body => %(
            :title("Web edit of {%edit<path>}"),
            :head($branch),
            :base<main>,
            :body("Edit suggested by ｢{%edit<editor>}｣ because ｢{%edit<comment>｣}"),
        )
    );
}

The Github documentation for the PR was particularly confusing because head, base, and body did not seem to be intuitive taken on their own.

Exiting the wood

I have documented my journey into and out of the woods using as little jargon as possible. My purpose in doing this was to describe it so I could understand it myself, and also in the hope that the gods of the internet (aka search engines) will guide someone else who may have a similar set of issues.

Naturally, the container is part of a bigger system, and it needs to be tested. But that raises more questions, so I will only be brief.

It is not so easy to test a websocket. My approach was:

to create a custom container for Cro,
add the service.raku program above.
The docker container is run locally (using Podman Desktop), so the websocket host is localhost.
A small html file with a websocket and a way to pass the data needed for the suggestion-box is loaded as a file into a browser.
A small Github repo is created with a file to edit.

Ryuu - a Japanese dragon

Richard Hainsworth — Tue, 30 Jan 2024 18:30:58 +0000

A follow up to the Welsh dragon.

Firing up another localisation

Steps to Ryuu

Comments on the Raku program

More generally about localisation of coding

If you want to make Ryuu better?

Firing up another localisation

In my previous blog about Y Ddraig, I created a localisation of the Raku Language in Welsh. During a recent conversation, someone mentioned there may be interest in a Japanese localisation, so I thought I would try the same techniques.

I do not speak or read or have ever studied Japanese. The localisation given below will be about as clunky and awkward as any can be. I imagine there may be some hilarious stupidities as well.

So to be clear, this article is about a proof of concept rather than a real effort to create a production-ready program.

However, it took me 40 minutes from start to finish, including setting up the github repo.

Since I like dragons, I named the Japanese cousin to Raku 'Ryuu'. It's a whimsy, not to be treated with much seriousness.

Steps to Ryuu

Basically I created a github repo, copied my existing Welsh localisation and changed CY to JA, and draig to ryuu.

Within the automation/ directory I used the translation technique explained for Welsh to create the JA file from the template. The translated.txt file needed some manual cleaning, because the English word for has multiple Japanese equivalents. I chose one more or less at random. In addition, Google translate did some strange things to the order of words and numbers in a line.

After adapting the files in the bin/ directory, and installing the distribution with Raku's zef utility, I ran tr2ryuu on the Raku program simple.raku.

A comment about my Welsh blog was that the program in Y Ddraig was not all in Welsh. And here the program is not all in Japanese.

Remember that the user-facing part of a program will be in the language of the user, in this case it is English. However, the coder-facing part of the program will be in the language of the coder. Below, the coder interface is in Japanese (or rather my ham-fisted attempt at Japanese).

The following is the result (which I put in a file called simple.ryuu):

私の $choice;
私の $continue;
私の @bad = <damn stupid nutcase>;
リピート {
    $choice = プロンプト "Type something, like a number, or a string: ";
    言う "You typed in ｢" ~ ($choice ~~ 任意(@bad) ?? "*" × $choice.文字 !! $choice) ~ "｣";
    与えられた $choice {
        いつ "dragon" {
            言う "which is 'draig' in Welsh"
        }
        いつ 任意(@bad) {
            言う "wash your mouth with soap"
        }
        いつ IntStr {
            言う "which evaluates to an integer ", $choice
        }
        いつ RatStr {
            言う "which evaluates to a rational number ", $choice
        }
        デフォルト {
            言う "which does not evaluate to a number "
        }
    }
    $continue = プロンプト "Try again? If not type N: "
} まで $continue 当量 任意(<N n>)

What is amazing to me is that when I ran ryuu simple.ryuu, the program ran without error.

Comments on the Raku program

The simple.raku program is obviously trivial, but what I wanted to show are some interesting Raku features. Note how I created an array of words with @bad = <damn stupid nutcase>;, and then later I tested to see whether an input word was one of the array elements.

The Raku idiom いつ任意(@bad) or in English when any( @bad ) compares the topic variable, in this case the input value, with each array element and creates a junction of Boolean results. The 'any' effectively or's the result to collapse the junction.

Junctions are not common in programming languages, so I thought if there would be problems, then it would be there. So I was surprised to find my Raku program works without error in another language.

More generally about localisation of coding

All the major coding languages are in English. There are, however, coders from all over the world, and the majority of those from non-English speaking nations would have needed to learn English before (or at the same time as) they learnt coding.

We are thus creating a new technological elite: those who can understand English (or some subset of it), and those who cannot. The more coding becomes an essential part of life, the greater the ability divide between coders (who speak English) and non-coders will become.

The aim of localising a programming language is to provide an entry into coding in a form that is more accessible to every human being, whatever their natural language.

However, the aim of this approach is not to eliminate English at every level of complexity, but to provide a sufficiently rich language for most normal coding and educational needs.

In addition, by having a canonical language (Raku, which is based on English) into which all localised languages can be translated, what we get is a universal auxiliary language together with a universality of being able to code.

Having a single auxiliary language means that a non-English speaking person writing in a localised coding language can translate the program with the problem into Raku, have a developer on the other side of the globe find the problem, and suggest a solution in code, then for that solution to be translated back into the local language.

Naturally, a person who wants to learn more about coding, or who needs to delve deeper into the workings of a module, will need to learn English. Learning wider to learn deeper is a normal part of the educational experience.

If you want to make Ryuu better?

Ryuu or however it should be called, absolutely is in need of Tender loving care. Please feel free to use the github issues or PR processes to suggest better translations.

At some stage, Ryuu will join the official Raku localisations.

Creating a new programming language - Draig

Richard Hainsworth — Mon, 15 Jan 2024 08:18:08 +0000

Actually creating a localization of an existing programming language in an existing human language

Introduction
Considerations
The plan for y Ddraig (the dragon in Welsh)
Constraints and first steps
Forwards into Draig and running
Completing the translation
Backwards to canonical form
Drawbacks

Introduction

Nearly all programming languages that are widely used in the world today have English as their base human language.

This means that a young person living in a non-English environment must first learn English (if only a limited sub-set of English), and then learn the skills needed for coding. This puts the majority of the humanity at a disadvantage.

Would it not be useful to create programming languages that use the script and words of human languages, but which compile into programs that will run with state of the art computer software?

Here is how I created a Welsh cousin of Raku, and I called it y Ddraig - or The dragon.¹

Considerations

There are some practical obstacles to creating any new programming language, and here are some of the ameliorating reasons why the Raku Programming Language is a good choice to base a new one on.

Different human languages use different writing systems and most need extra letters not covered by the ASCII set
- The Unicode system has eliminated the problems of displaying and storing different writing systems.
- The Raku language has Unicode support at every level. Every number, operator, keyword, function etc can be written with Unicode symbols. There are a very few exceptions, such as ;, ,, and {}.
Different operating systems
- There are at least three major operating systems widely used in the world: Window, PC Linux, and Mac OS.
- Raku runs on all three
All professional programmers are proficient in English, and so can answer questions about program errors in English. The number of programmers speaking Welsh is quite small, and the same would be true for many other human languages.
- This is precisely the problem we are trying to resolve: making the coding profession a possibility for humanity as a whole. But there is a vicious circle, professional programmers work in English, so how can they help a person in another language?
- The circle can be broken if the interface language, which is the one that programmers work in, can be varied, whilst the code that is run on a computer is completely independent of the interface language.
- If the interface language can be easily changed, then a program coded in Welsh, but with a problem, can be easily translated (whilst continuing to work as before) into English. The problem can be resolved, corrected, and the solution changed (still working) back to Welsh.

The plan for y Ddraig

Whilst the plan is to create y Ddraig as a language that can be used with as little English as possible, there are several stages:

First is to create a localization (L10N) of Raku, or a module called L10N::CY.
- Inside a Raku program, all that will be needed for a completely Welsh program is for the first line to be use L10N::CY;
- All subsequent lines will be in Welsh, but will compile and run as a normal Raku program.
- The program can be easily translated into English, and English Raku programs translated in to Welsh. Simple utilities are given below to do this.
Next, the operating system has to be adapted so that a executable called draig is available, which will also mean that in a graphic interface (GUI), double clicking on a file with a file-extension of .draig will run Raku with the L10N::CY module already loaded. This is trivial.

Constraints and first steps

For personal reasons, I stopped using Windows on my PC, and I use Ubuntu Linux exclusively. So, where there are terminal sessions, I shall be showing how I created Y ddraig using a Linux terminal.

Since Y ddraig is a Raku cousin, or technically a Raku localization, the Raku language needs to be installed. In addition, it needs to be a version of the language released after December 2023. Information about the installation of Raku, and its package manager zef, can be found on the Raku website.

The first stage is to create the L10N::CY module. It is simply a normal Raku module, which is then installed with the zef package manager.

Raku module development is conventionally done by creating a github repository. Working with git is quite simple for the basic functionality, but there is a long learning curve when working with others. But none of that is the topic here.

Elizabeth Mattijsen, who is responsible for all this Raku internationalization magic, has created a template internationalization module for the Klingon language (yep: aliens get to be the first to use localizations of a Terran computer language)².

So I git cloned the Klingon, and created a github repo for the Welsh. My git nick is finanalyst, so here's the terminal code lines:

git clone https://github.com/lizmat/L10N-TLH.git rakuast-L10N-Klingon 
git clone https://github.com/finanalyst/rakuast-L10N-CY.git rakuast-L10N-Welsh

In the following, I shall call Elizabeth's repo, the Klingon repo, and mine, the Welsh repo. If you want to create your own language, the convention being followed is to name the language according to an ISO 639-1 supported language code, at least for the foreseeable future. You should also think of an filename extension (like .draig here) for programs in the new language (Raku cousin).

The two critical parts of the module are update-localization, and a root text file which we will call the localization map. It should be named by the language code. Here it is called CY for Cymraeg or the Welsh language, for Klingon, it is TLH.

The update-localization utility in from the Klingon repo looks for a repo root directory file with 2 or 3 upper case characters. This is taken as the localization map and is automatically converted into all the magical modules.

The biggest step is to translate the terms to be stored in CY. The template for the localization map can be found at Github Raku localizations. To get this as a local text file, I used the following terminal code to download the template in to my working directory.

curl 'https://raw.githubusercontent.com/Raku/L10N/main/TEMPLATE' > CY

The pristine form of CY contains a few lines of comment (starting with the characters '# ', note the space), and then a number of sections starting with

# KEY        TRANSLATION

Within each section there is a key and then an English Raku keyword, eg.

#adverb-pc-delete         delete

Note that it has been commented out with single #. This means that the update-localization utility will ignore the line.

Now comes the translation part. Each significant commented line (a line with # and no space at the start) has two parts: a KEY and a TRANSLATION, with some spaces between them. The translation process is to substitute the English Raku keyword with the Welsh word, and remove the #. For example, the first significant line becomes

adverb-pc-delete                 dileu

When starting the translation process, and to see how the system works, it is sufficient to translate a minimum number of keys. (Eg., for the Draig program below, I only need eleven words.)

Once I have enough key words for the program, all that is needed is to run ./update-localization. This then creates a directory tree under lib/.

Forwards into Draig and running

Here is a short program in Raku (English cousin), which we store in a file called 'simple.raku' in the root directory of the repo.

my $choice;
my $continue;
my @bad = <damn stupid nutcase>;

repeat {
    $choice = prompt 'Type something, like a number, or a string: ';
    say 'You typed in ｢' ~ ( $choice ~~ any( @bad ) ?? '*' x $choice.chars !! $choice) ~ '｣';
    given $choice {
        when 'dragon' { say "which is 'draig' in Welsh" }
        when any( @bad ) { say "wash your mouth with soap" }
        when IntStr { say "which evaluates to an integer ", $choice }
        when RatStr { say "which evaluates to a rational number ", $choice }
        default { say "which does not evaluate to a number "}
    }
    $continue = prompt 'Try again? If not type N: ';
} until $continue eq any(<N n>) ;

Try running it in a terminal where the working directory is the root directory of the repo, thus:

raku simple.raku

If you input some words, it will tell you the input is a string, if you input something naughty (well only one of the three words 'damn stupid nutcase'), you will get another response, and then there are responses depending on whether the number is an integer or a rational.

The code uses 11 keywords, which I translated and put into CY. Obviously, there are many strings that form the user interface, and these are hard-coded in this program in English. We are concerned at the moment with the infrastructure keywords that form the programming language.

Now lets translate the Raku program using a simple Raku utility called tr2draig.
We shall specify here that the Raku program is of the form somename.raku and that we want a Draig program of the form somename.draig.

The utility is the following Raku script:

#!/usr/bin/env raku
sub MAIN(
  $filename where *.IO.f  #= source file to be localized to Welsh
) {
    $filename.IO.extension('draig').spurt: $filename.IO.slurp.AST.DEPARSE("CY")
}

Breaking the program down, #!/usr/bin/env raku is standard for a script with execute permission.

$filename where *.IO.f #= ... is a nice Raku idiom for a program called from a terminal. The program expects a string that names a file. It checks that the filename exists and is of type 'f'. If not, then an error message will be provided from the comment following #=.

$filename.IO.extension('draig').spurt: takes the filename, creates a new file with the extension 'draig' replacing the previous extension (which was 'raku'), then spurts text into it, the text it uses being generated by the expression after the :.

$filename.IO.slurp.AST.DEPARSE("CY") takes the filename (which has extension 'raku'), makes it into a filehandle, slurps (sucks) in the text that is in the file, parses the text as a Raku program into an Abstract Symbol Tree (AST), and then deparses the symbol tree using the new Welsh keywords into a new program with Welsh.

For reasons related to distributing Raku software, I have placed the utility in the bin/ directory. There are two ways to get a copy of these files, either by creating a clone of my Github repository (the url is given above), or by installing the Raku distribution, as zef install "L10N::CY". If zef is set up in a typical way, then the utilities below can be run without specifying the path.

The translation utility is run like this

bin/tr2draig simple.raku

This produces a file simple.draig, which contains

fy $choice;
fy $continue;
fy @bad = <damn stupid nutcase>;
ailadrodd {
    $choice = prydlon "Type something, like a number, or a string: ";
    dywedyd "You typed in ｢" ~ ($choice ~~ unrhyw(@bad) ?? "*" x $choice.golosg !! $choice) ~ "｣";
    a-roddwyd $choice {
        pryd "dragon" {
            dywedyd "which is 'draig' in Welsh"
        }
        pryd unrhyw(@bad) {
            dywedyd "wash your mouth with soap"
        }
        pryd IntStr {
            dywedyd "which evaluates to an integer ", $choice
        }
        pryd RatStr {
            dywedyd "which evaluates to a rational number ", $choice
        }
        rhagosodedig {
            dywedyd "which does not evaluate to a number "
        }
    }
    $continue = prydlon "Try again? If not type N: "
} hyd $continue eq unrhyw(<N n>)

Now we want a way to run draig programs. The easiest way is create another Raku program draig, which we place in the bin/ directory. bin/draig has the following content:

#!/usr/bin/env raku
sub draig(*@_) {
    %*ENV<RAKUDO_RAKUAST> = 1;
    %*ENV<RAKUDO_OPT>     = '-ML10N::CY';
    run $*EXECUTABLE, @_;
}

multi sub MAIN() {
    draig
}

multi sub MAIN(
  $filename where *.IO.f  #= source file to be run in Welsh
) {
    draig $filename
}

Here's a gloss of the program:
sub draig(*@_) {... This is a helper subroutine called later. It sets up environment variables, and preloads the localization module, before running Raku with the Welsh keywords.

multi sub MAIN() runs the sub draig (above) when no program is given. This puts the user into a REPL, where statements can be input directly, parsed and run immediately. However, draig will run using the Welsh keywords.

multi sub MAIN( $filename where *.IO.f #= source file to be run in Welsh ) handles the case when draig is given a filename. As explained above, the filename is tested for existence.

Now try running bin/draig simple.draig in a terminal.

If the RakuAST-L10N-CY distribution has been installed with zef, then all you will need is draig simple.draig.

The running code produces exactly the same output as the English Raku program. The user interface output is still in English, and for completeness, I should translate all of the text strings to Welsh as well.

Completing the translation

At this point, we can translate any English version of a Raku program into a Draig program, and draig will run it, but only if the Raku program uses the 11 keywords I translated.

In order to create a full localization, all of the Translation values need to be converted to Welsh. The first step (and I really must re-emphasise it is a first step) is to use an automated translation tool. A correct localization will need first-language Welsh speakers to go through the CY file and correct the translations.

At the time of writing, the localization has not been properly verified, so it has not yet been added to the official Raku localizations.

For the automated translation, I have created the directory automation/. I again downloaded the TEMPLATE into a CY file in the automation/ directory.

I have written some automation helper utilities, namely:

find-untranslated, takes a CY file and splits it into two new files, with line numbers at the start of each line to help match later. One file is partial.txt with the starting key and comment lines, and the second file is to-be-translated.txt. Both contain approximately 700 lines.
combine-translated, takes partial.txt and another file translated.txt (see below) to create a new CY file.

Next I copy/pasted the lines for translation (from the file to-be-translated.txt into Google's translate to Welsh page. The operation took a couple of copy/pastes due to size limitations, but the text is not overly large.

The translated text can be copied straight back to a new file (translated.txt), and then recombined with partials.txt to create CY.

Backwards to canonical form

As mentioned above, suppose a Welsh-speaker using y Ddraig
runs into a programming problem, a syntax error or logic not working as the programmer assumes. An English speaking programmer will probably not be able to help.

But ... .draig program can be retranslated back to the canonical form of Raku. This is done by a utility called tr2raku. It is almost the inverse of tr2draig, but instead of replacing the file extension .draig with .raku, we add it on to the filename so that its clear it is a canonicalisation of a Raku cousin.

The utility bin/tr2raku contains the following contents.

#!/usr/bin/env raku
sub MAIN(
 $filename where *.IO.f  #= Welsh source file to be turned to canonical form
) {
   $filename.IO.extension('raku', :0parts).spurt: $filename.IO.slurp.AST("CY").DEPARSE
}

The difference can be seen that the language signifier (CY) is a parameter to the AST method, rather than the DEPARSE method.

There should be no reason why this recipe cannot be applied to Mandarin, Hindi, or Japanese.

Drawbacks

The problems stem from the development history of Raku. Error messages are in English, and so Raku cousins, like Draig, will have English error messages.

The problem is not insurmountable, but it will take a lot of translator hours.

Another problem is that helper modules, for example, JSON::Fast, which imports/exports structured data from/to .json files into Raku data structures. The module has two main methods to-json and from-json. These names are set by the module, not by Raku.

A program in y Ddraig will be able to access all Raku modules without restriction, but it will need to use the canonical (English) names.

However, if many Raku localizations come into being, and a user base for them develops, these are all soluble problems.

Footnotes

1

A reader may wonder why the language is Y ddraig, but draig is given in dictionaries as the translation for dragon. Well ..., draig is a feminine word, and the definite particle Y triggers a mutation in the next feminine word, so d mutates to dd.

2

My next project is to create a localization with Egyptian hieroglyphs

RakuDoc revision open to comment

Richard Hainsworth — Mon, 31 Jul 2023 23:00:00 +0000

The second stage in the process to update RakuDoc is now over and the third (GAMMA review) stage is starting. In order not to repeat some history, please take a look at Revising Rakudoc.

An online version is available of the proposed RakuDoc language.

The whole of the Raku documentation suite is written in RakuDoc.

Improving on a good design

About half of the original design ideas outlined in S26 were documented in current POD6. Some of the ideas were available, but not documented. Some instructions were not realised at all.

It should be remembered that RakuDoc is parsed by the compiler (eg. Rakudo) as part of a Raku program, and is then rendered by the renderer (eg. Raku::Pod::Render) into (for example) HTML. When I use the word 'implemented', I mean that a RakuDoc instruction is properly parsed and rendered. Some of the instructions defined in S26 were parsed by Rakudo, but not rendered, and some were not parsed properly or at all, so could not be rendered.

The revision process has therefore identified and rectified the parsing deficiencies, and identified the rendering flaws. RakuDoc is correctly parsed only on the most recent versions of Rakudo, which at the time of writing has yet to be released. Raku::Pod::Render still does not handle RakuDoc in its entirety.

Two use cases

It became clear that the RakuDoc serves two inter-related use cases:

Documenting code for developing and maintaining software.
- explanations about variables, methods, etc.
- another program that uses the software as a dependency should be able to access these explanations
- data which should be included with the software but not be hard coded into it, for example, a unit test needs to populate a data structure with test data.
Documenting the software for use by another user.
- Good documentation should have tutorials, code snippets, headings, Tables of content, etc

Tables

RakuDoc had a simple table markup, which is very similar to the Markdown syntax. It worked, but the simplicity of the syntax was at the cost of flexibility.

Looking around at other ways of specifying a table, we identified two paradigms (there may be more), namely the one used by HTML and the one used by the GTK grid widget. Both of them allow for cells that span more than one column or row, and both allow for embedding (eg. a table inside a cell of a table).

After several iterations, a new procedural model was created and rendered. The design allows for spanning and embedding, but it also allows an author to specify a table row by row, or column by column, or even using a mixture of both.

An example showing a markup using both rows and columns can be seen in the online draft.

Semantic blocks

A semantic block is a section of text that should be easily available to another software tool, or can be moved around the final document.

For example, a section on the authors of a document (including contact or affiliations) is most easily written at the top of the document, but often it is better to place the information towards the bottom of the text.

This is done by creating a semantic block (simply by making the calling the block in uppercase letters). The block can be hidden from view by adding the metadata option :hidden. All the data is placed in a special structure.

The rendered text can be placed in the document later using the P<> instruction, or it can be accessed by another tool that may only be wanting the VERSION or LICENSE.

More metadata options

One of the strengths of RakuDoc is the ability to add optional metadata to blocks of text.

The new version of the defining document explains this concept in more detail. Metadata options are optional, with reasonable defaults being assumed. This means that a short form of the block is sufficient in most cases.

In the description above, the option :hidden was mentioned. Another example, is :caption. Suppose you want to write a semantic block called =AUTHORS at the start of the document, but you want for it to appear later in the document as Article authors, then you could specify it as follows:

=for AUTHORS :caption<Article authors> :hidden
A. N. Writer, socMedia nic @psuedonym
M. Z. Orator, socMedia nic @politician

Article text continues

Pages later

P<semantic: AUTHORS>

It is possible to include a link L<for reference see | #A very long title somewhere in the text> where the text on the right-hand side of the | is a heading. However, this can become tiresome if you want to include several links to the same place.

So, a metadata option :id can be included in a heading. This allows you to do the following:

=for head3 :id<lnk>
How to correctly link to other places in a manual

Pages of text

Properly linking is important, L<see for example|#lnk>

Doing things in line

RakuDoc has instructions for block level text, such as headings, paragraphs, code.

Typically blocks will be included in the Table of Contents.
It also has markup instructions that work in line, and which do not (typically) affect the ToC.

For example, a simple markup instruction is C< text >, which renders like text. I have used the Markdown equivalent here. In RakuDoc, everything between the C< and > is verbatim and styled differently to normal text, just like the Markdown code quotes. However, RakuDoc also has V< text > which treats everything inside the angle brackets as verbatim but does not style it differently.

A new markup instruction in RakuDoc is M< text | metadata>. A renderer will place the text in the rendered text, but will also provide a mechanism for the user to take the metadata and provide new functionality. For instance, M< fa-copy | font awesome v5 > could be interpreted to insert the font-awesome icon called fa-copy into the text. Or M< Buy now | PayPal, database-id > could expose the API for the PayPal payment platform.

How not to be confusing

RakuDoc is inherently customisable. It is also designed to be output neutral (although at the moment HTML is the most common output form). Semantic blocks can be invented within a document, and a renderer can allow for other user-defined blocks and markup instructions to be created.

However, RakuDoc is specific about naming rules. A built-in block must be all lower case, and renderers should not allow user-defined blocks to use all lower case. A semantic block is all upper case. And a user-defined block must have at least one upper-case letter and one lower-case letter.

All markup instructions, which are inline instructions, must be a single Unicode character with the property UPPER. Built-in markup instructions are the ASCII characters and Δ. All other codes can be used.

The naming rules have been created to ensure that even if a user-defined block or markup becomes popular, it is not a part of the RakuDoc standard. Renderers are only required to implement the RakuDoc standard, and may render other blocks, or not.

Wrapping up

These are some of the interesting additions to RakuDoc that are being proposed. There are more.

Since the Gamma review stage is now underway, it is almost certain that there may be more changes because the revision is now open to the Raku community for comment and requests. Discussion is open both for the language design and for the explanation of the design.

As might be admitted, community requests for changes to the overall design will face significant resistance from the main authors in order to maintain backwards compatibility with the previous version of RakuDoc, and the integrity of the underlying paradigms. New block or inline instructions will be more readily considered, but requests for examples, explanation, and greater clarity will be very much appreciated.

Revising Rakudoc

Richard Hainsworth — Fri, 30 Jun 2023 23:00:00 +0000

In the earliest days of Raku, Damian Conway specified a documentation markup language to accompany it. Since it was modeled on Perl's POD it was called <sound of trumpets and dramatic pause> POD6.

The Specification of POD6 (S26) was mostly incorporated without much extra explanation in the documentation suite. In this way, the description of POD6 was itself was an illustration of many of the features it documented, and some that it did not document.

Since Raku is defined by its test suite, and not its documentation, there were other details of POD6 in the tests that were not documented, even in S26.

Raku developed and morphed, but POD6 remained. The tooling for rendering the documentation sources needed updating, and the documentation site had to be modernised.

Upgrading the renderer

A project of mine was to upgrade the basic renderer that would transform POD6 to HTML, but allow for developers to customise the templates for each type of POD6 block type. (The first Pod::To::HTML renderer hard-coded representations of POD6 markup, eg. B<this is bold> was <strong>this is bold</strong> and could not be changed.)

It turned out that S26 allowed for much more than had been included in the first documentation sources, including custom blocks and custom markup.

The project to upgrade the original HTML renderer morphed into Raku::Pod::Render, and transforming a directory full of individual documentation sources into an interlinked and searchable set of documents required another layer of tooling Collection. For example, collecting together all the pages that can be grouped as tutorials, or reference, or language, and creating a separate page for them automatically.

I covered these two projects in a presentation to RakuCon 2022.

Some of the original ideas in S26 had not been implemented, such as aliases and generic numbering. Other ideas had become outdated, such as a way to specify document encoding, which is now solved with Unicode.

In addition, RakuAST (see RakuAST for early adopters ) is on the horizon, which will radically change the speed of documentation processing.

There are also two implementations of POD6, one in Raku and one in Javascript, namely Alexandr Zahatski's Podlite.

Introducing Rakudoc

This was an ideal time to revisit POD6 and recast it into Rakudoc - new name for the markup language, and its new file extension ".rakudoc".

I was invited to the first Raku Core Summit and I put together a presentation about the changes I thought needed to be made based on my own experience, but also using comments from other developers.

We came to a number of consensus agreements about the minimal changes that were needed, and some extra functionality to handle new questions, such as documentation versioning.

It was also clear that Rakudoc (aka POD6) has two separate parts: components that interact closely with the program being documented, and components that will be rendered separately into HTML (or an ebook). The documentation file needs to make this clear.

I have now written the first draft of the revision and the documentation file that encapsulates it. An HTML version can be found at new-raku.finanalyst.org/language/rakudoc, alongside the old documentation file and the simple table implementation. I am planning future blogs to describe some of the proposed revisions.

However, none of the revisions will break existing POD6, so Rakudoc should be backwards compatible with POD6. The version at new-raku is a VERY early first draft, and it will go through several review stages.

The first Raku Core Summit was organised by Elizabeth Mattijsen and hosted by Elizabeth and Wendy at their home. It was a really good meeting and I am sincerely grateful for their generosity and hospitality. The summit was also supported by The Perl and Raku Foundation, Rootprompt, and Edument.