Enumerating in RakuDoc v2

We needed a legal document with numbered Articles, and the document needed to be in MarkDown for github issues, and in html, so naturally the source should be in RakuDoc.

Although numbered headings and items were specified in RakuDoc v1 (aka POD6), they were never implemented. In RakuDoc v2 there was progress, and together with some tweaking of the numitem templates, it was easy to write the RakuDoc source.

However, when we were revising RakuDoc, Damian Conway had some extra ideas about generalising enumeration, including ideas about adding alias definitions so that the numbered block could be referenced later in the text.

Since it had been so easy to tweak the numitem templates, I thought it would be easy upgrade the specification of RakuDoc v2 to get these generalisations. Was I ever so wrong!!!! Ask a genius with decades of language design experience for a nice design and you get an effusion of ideas and extensions that make RakuDoc better than any editor I have ever used, but with a simplicity that makes it easy for a document author to understand.

Another result of the redesign is to make the underlying specification of RakuDoc much clearer, something I will cover later.

During the development process, my daughter mentioned that her friend had just finished a PhD dissertation and was complaining about how much time she needed to spend reformatting the text because the numbering kept getting out of sync. The problem with most editors is that most of the effort goes on perfecting the user-facing interface, while the underlying format is created ad hoc, and enumeration is an addition. Getting the underlying structure right will make subsequent rendering easier.

Just as the design was ending and the renderer passing most tests, I casually mentioned citations would be a good extension. The "standards" for citations number in the thousands! .oO(The Jabberwocky ) But Damian snicker-snacked his vorpal blade, reducing the monstrous tangle to something easier to use. RakuDoc v2 now has a =citation block and Q<> markup to insert quoted citations. I will cover these additions in the next blog (as in: when I get the renderer to work with the new ideas).

Back to enumerating RakuDoc, here are some examples to illustrate the new functionality.

1. Suppose you have a formula (or code or map or table or item in a list) and you want to number it, and then reference the number in the text? Then another formula gets added into the text before, well you would want the references to update as well.

2. Suppose you have tables that you want to be enumerated separately from headings? But also by preference you want them to be enumerated in sequence with headings? That is the prefix to the table enumeration is the heading number.

3. Suppose you want Chinese, Roman, or Bengali numbering?
   AND suppose you also want the numbering to be a mix of numerals and other characters, such as brackets? This is a common requirement in legal documents.

4. Suppose you want Arbitrary words before after or in between the numbers? For example, Article 1., Article 2. etc.

5. Suppose you want to have the paragraphs numbered? Sometimes you might want the paragraphs numbered in sequence from the start of the document, and sometimes you want the numbering to restart after a new heading.

6. Suppose you want to have the Tables and the Formulae numbered in sequence? Or perhaps, for a section that explains some aspect of one formula, you want to number the formulae separately, but then return to the original sequence after that section?

The updated RakuDoc v2 allows for all of these possibilities, while also providing for sensible default option values.

Moreover, RakuDoc allows for custom blocks, eg. LeafletMap - a map block that exposes the marvelous Leaflet library. Now it is an automatic part of the specification that prefixing a custom block with num (numLeafletMap) will number the caption of the block.

An HTML rendering of the new RakuDoc v2 specification containing the enumerated functionality can be found at RakuDoc enumeration branch

Understanding the RakuDoc paradigms

In the specification of RakuDoc, there are discussions about directives, blocks, metaoptions, and a section on =config.

Since the syntax for a directive is almost the same as the syntax of a block, it is not immediately apparent what the difference is. As we developed the generic enumeration, it became much clearer what the difference is.

Furthermore, the older specification covered multi-level headings and items. This functionality is now extended to all blocks, so =numtable2 or =numcode3 will be numbered in parts from the previous instance of =table (the equivalent of table1) or =code. This means that we need to carefully distinguish between the block base (eg., table or code), and the block level (eg., 1, 2, 3). Together the base and the level create a blocktype. These distinctions, although implied, were not so clear in the older specification.

Let's return to the =config directive. The first parameter of =config is the name of a blocktype. Note that the num prefix is not a part of the blocktype and only indicates that the enumeration associated with the instance needs to be rendered. Consequently, the presence or absence of 'num' in the config parameter has no significance. The next =config parameters are all, and may only be, metadata options.

The function of the =config directive is to distribute the named metadata options, and their values, to the named blocktype. In addition, the same metadata option can be specified on a blocktype instance (using either the =for or =begin directives), in which case it takes precedence over the option in the config. A consequence of this paradigm is that each metadata option has to have a semantic significance in the context of a blocktype instance.

One of the design aims was to give a document author the choice of restarting the enumeration for some blocktype when another blocktype is encountered. For example, the original specfication for =numitem included the idea that whenever a sequence of =numitems was encountered, they would form an ordered set, and that if another block, such as a paragraph, was encountered, the enumeration would be restarted.

This means that the occurrence of a =head restarts the item counter. This cannot be controlled within the handler of a block. Consequently, any option affecting block counting needs to be distinct from the rendering of the blocks themselves.

After some design iterations, a new directive called =counter was introduced. A directive, as opposed to a block, affects all subsequent blocks in the RakuDoc source, within the same scope. A block, and the metadata options operating on the blocktype, only affects its immediate contents.

Further, it became clear that a block and its counter were different objects, although for simplicity they have the same name. For most needs, an author does not need to know that a counter and a block are different, except that when a new counter is needed, the difference becomes necessary. In a similar way, for many purposes it doesn't matter whether a number is an integer or a string, except when it does. Raku allows for things to become complicated when the author needs it to be.

How to experiment with enumerations?

To make it easier to experiment with RakuDoc and the enumerated functionality, I have developed a Docker image called browser-editor. It is based on an Alpine image and contains raku, Cro, and the latest version of Rakuast::RakuDoc::Render (currently not yet in the fez system).

Using podman, the image can be put into a container and run locally on Linux based systems, thus:

podman pull docker.io/finanalyst/browser-editor:latest
podman run -d -v .:/browser/publication --rm -name rb docker.io/finanalyst/browser-editor:latest

For Mac silicon, a small change is needed to indicate the platform inside the docker image is based on Linux, thus:

podman run -d -v .:/browser/publication --rm --platform linux/amd64 --name rb docker.io/finanalyst/browser-editor:latest

In both cases the container is given the arbitrary name rb, and so when it is time to stop the container, the following is sufficient:

podman stop rb

The directory that the container is started from will then linked to the container's /browser/publication directory, and changes and new RakuDoc source files will be saved in that directory.

A RakuDoc source can then be edited and the HTML rendering is created on the fly, by pointing a browser at (setting the browser URL to) localhost:3000.

You should see something like the following in the browser:

Two frameworks exist: one needing no internet connection - a minimal single file rendering, and another that uses plugins to expose some useful third-party libraries and the more sophisticated Bulma CSS framework. The choice is toggled using the 'online' button.

Try selecting 'online' and copying in the following test source by Damian Conway to see some different effects.

=begin rakudoc
=TITLE Taster source
=numitem One
=numitem Two
=counter item :restart(42)
=numitem Three
=counter item :restart
=numitem Four
=counter item :prefix<head3>
=numitem Five
=counter item :!restart
=para ???
=numitem Six

=numpara
S<Now is the winter of our "no content"
Made glorious summer by Camelia’s bloom;
And all backlog that lour’d upon our docs
In deep commits is buried, link’d, and tagged.
Our nightly builds now wear triumphant green,
Our failing tests to passing smiles are turn’d;
Grim sighs of “ere next Yule” have chang’d to grins,
And hacker dread to blog posts boldly strung.>

=numpara
S<But I—long nurs’d on RFCs and hope,
Deform’d by specs that shifted as I read,
Unfit for idle scripts or stable sleep—
Am set, since long delays are now no more,
To ship Raku...and break the world anew.>

=numcode
$x = any <1 2 3>;

=for numcode :caption<Same thing, just labelled>
$x = any <1 2 3>;


=for numformula :caption<This means nothing!>
x^2 = y_j + \sum z_i

=for numformula :caption<This means nothing!> :alt< xH<2> = yJ<1> + E<GREEK CAPITAL LETTER SIGMA> zJ<i> >
x^2 = y_j + \sum z_i

=for numformula :alt< (1+x)H<n> = E<GREEK CAPITAL LETTER SIGMA> H<n>CJ<i> xH<i> > :caption<This is actually right>
(1+x)^n = \sum_{i=0}^n {n \choose i} x^i

=for numinput
Hey type something:

=numoutput
You typed: hsdkhldkhskdhasd

=counter numtable :restart(33456)
=begin numtable :caption<Encryption table> :form<Example %R: %C:pc>
=row
=cell A
=cell B
=cell C
=row
=cell D
=cell E
=cell F
=end numtable

=end rakudoc

This should look like the following (or at least the top part).

The example shows some features like numbering paragraphs, code examples and tables. But you will also see the difference between the two rendering contexts. The online version has access to an online latex rendering resource, so the formula are nicely rendered, whilst the off-line version only renders to the raw formula or a character-based alternative - if provided.

Coding the examples in the introduction

Adding an alias to a block

Suppose we want to enumerate a code sample and then refer to it later, we can do the following:

=begin rakudoc :!toc
=TITLE Enumerations
=for numcode :numalias<SAY_EX> :lang<raku>
my %h = <one two three> Z=> ^3;
say %h;

=numoutput {one => 0, three => 2, two => 1}

=for numcode :numalias<PUT_EX> :lang<raku>
my %h = <one two three> Z=> ^3;
put %h;

=for numoutput                                                                                                                                    
one     0
three   2
two     1

The difference between A<SAY_EX> and A<PUT_EX> is that C<say> and C<put> use different methods to convert the C<%h> structure into printable strings.
=end rakudoc

In the browser-editor image, we will get an HTML rendering something like:

The =begin rakudoc :!toc and =end rakudoc are used to top and tail the RakuDoc example above because the HTML renderer expects a complete Raku program, and a RakuDoc source (currently) needs the RakuDoc to be within a =rakudoc block. The :!toc switches off the automatic Table of Contents that the HTML utility of the Rakuast::RakuDoc::Render distribution generates. (Try removing :!toc)

Prefixing an enumeration with another counter

Suppose we want out tables to have enumerations that align with the enumerations of the headings. So this means we are requiring a different sort of behaviour from the counter associated with the table block base. Consequently, we need to use the =counter directive. Also note that we are prefixing with the head2 block, which has an implicit prefix of head1.

=begin rakudoc :!toc
=TITLE Prefixes
=counter table :prefix<head>
=numhead First title
=numhead2 Sub first title
=for numtable :caption<First table>
| one | two |
=for numtable :caption<Second table>
| three | four |
=numhead Second Title
=counter table :restart(6)
=for numtable :caption<Third table>
| five | six | seven | eight |
=for numtable2 :caption<Subordinate table>
| nine | ten |
| nine | ten |
=end rakudoc

Comments:

• A table must have some content, and the 'visual' table format is the minimum possible
• In order to make the prefix numbering a bit clearer, I manually restarted the table counter to 6 before Third table.

Enumerations with different numbering systems

To illustrate the multi-lingual ability of RakuDoc (and Raku in general), lets use Chinese, Roman and Bengali for multilevel headings.

=begin rakudoc :!toc
=TITLE Some non-Arabic numerals
=config head :form< %Z %D >
=config head2 :form< ｢%Z｣%R %D >
=config head3 :form< ｢%Z｣%R｢%B｣ %D >
=numhead First title
=numhead Second title
=numhead3 Sub-sub-head one (implies the sub-head)
=numhead3 Sub-sub-head two
=numhead2 Sub-head first explicit
=numhead2 Sub-head second explicit
=for numformula2 :form<%T %Z.%R. >
\begin{align*}
\sum_{i=1}^{k+1} i^{3}
&= \biggl(\sum_{i=1}^{n} i^{3}\biggr) +  i^3\\
&= \frac{k^{2}(k+1)^{2}}{4} + (k+1)^3 \\
\end{align*}
=end rakudoc

This will produce an HTML rendering a bit like

Including arbitrary text

The :numform option is very flexible and in fact the brackets in the previous example are arbitrary text. The following RakuDoc source

=begin rakudoc :!toc
=TITLE Arbitrary text in enumeration
=config head :form< Article %N. %D >
=config head2 :form< Article %N.%r. %D >
=config head3 :form< Article %N.%r｢%Z｣ %D >
=numhead First title
=numhead Second title
=numhead3 Sub-sub-head one (implies the sub-head)
=numhead3 Sub-sub-head two
=numhead2 Sub-head first explicit
=numhead2 Sub-head second explicit
=end rakudoc

will produce something like:

Numbering paragraphs

In RakuDoc, paragraphs as in all text editors, are strings of text ending in a blank line or a new block (for RakuDoc). There is no need to mark a paragraph block, but it will be equivalent to a block marked =para.

By default, each paragraph is numbered from the start of the document. In order to see the enumeration, the paragraph does need to be started with a =numpara.

For this example, we want to number paragraphs in each section marked with a heading. So we need to tell the counter which conditions it is restarted by; in this case after every =head (by default a bare block name has a level of 1). There are both :restart-after and :restart-except-after conditions, but that's all we'll say about them here.

The RakuDoc source is

=begin rakudoc :!toc
=TITLE Restarting after headings
=counter para :restart-after<head>
=head First heading
=numpara First line
=numpara Second line
=numpara Third line
=head Second heading
=numpara First line 2
=numpara Second line 2
=head2 Subordinate heading does not restart the paragraphs
=numpara Third line 2
=numpara Fourth para
=end rakudoc

The source will yield something like:

Custom counters and block scopes

We want to number code and formulae using the same counter, but also we want to explain some formula with a special numbering that is then forgotten.

Since all =config and =counter statements are scoped, all we need to do to get the special numbering is to create a =section, which introduces a new block scope. The previous config/counter options continue, unless explicitly overridden.

However, it would not be useful if upon exiting a section, the numbering of a counter reverts to the value before the section. So there is a subtle distinction between the counter and the value of the count it contains. The counter options, such as its prefix, and when it is restarted, are scoped, but the value of the counter is not scoped. In order to reset the counter value, the counter has to be reset.

RakuDoc v2 clarified the idea of custom blocks. Previously it was implied that developers could have their own blocks, but in V.2 the difference between built in and custom blocks was clarified: custom blocks must contain a mix of upper and lower case letters (more precisely characters with the Unicode properties Lu and Ll).

The enumeration options use this idea by creating custom counters. Since this is something that managed by a block, the :counter option is provided to the appropriate blocks using a =config directive.

=begin rakudoc :!toc
=TITLE Using custom counnters
=config formula :counter< FormulaeTables >
=config table :counter< FormulaeTables >
=numhead First title
=numtable 
| one | two | three |
=numtable
| more | tabula data |
| 5 | 6 |
| 7 | 8 |
=numformula E = mc^2
=numformula e^{\pi i} = -1

=begin section
=config table :counter< Derivation > :form<Aside: %T %N>

=numhead2 Some explanation
=numtable
| this | data | is | more | detailed |
| 1 | 2 | 3 | 4 | 5 |
=end section

=numhead Continuing
=numtable
| here | we | resume |
=end rakudoc

This will produce something like

Afterword

We have only just completed the revised specification, and the Renderer has only just been developed. It is inevitable that there will be flaws, and the styling choices may not be liked by everyone.

Even so, we think this new upgrade of RakuDoc v2 will prove to have a much wider applicability than just a documentation aid to Raku programs.

The docker image allows anyone to experiment with the new RakuDoc functionality immediately without needing to install the whole Raku / Cro / Rakuast::RakuDoc::Render stack and some dependencies, such as Dart Sass, locally.