DEV Community: James Carlson

Implementing MicroScheme in Elm

James Carlson — Sun, 12 Mar 2023 13:53:36 +0000

I'd like to report on a little weekend project, namely to implement an interpreter for a small subset of Scheme using Elm. Here is a screenshot of the command-line interface:

The code (open source, of course!) is on Github with some
documentation there also. The code base, as outlined below, is quite small. I intend to keep it that way, since the goal is educational (I am learning a lot). I'll concentrate on a few core features, e.g., a proper implementation of environments and let-bindings.

--------------------------------------------------------
File                                               code
--------------------------------------------------------
src/MicroScheme/Eval.elm                            92
src/MicroScheme/Frame.elm                           64
src/MicroScheme/Parser.elm                          56
src/MicroScheme/Interpreter.elm                     43
src/Main.elm                                        41
src/MicroScheme/Environment.elm                     19
src/repl.js                                         19
src/MicroScheme/Expr.elm                            13
src/MicroScheme/Init.elm                            10
-------------------------------------------------------
SUM:                                               357
-------------------------------------------------------

There is more detail in the docs linked above, but perhaps just a few words more. The current state of the data type for Scheme expressions is

type Expr
    = Z Int
    | F Float
    | Str String
    | Sym String
    | L (List Expr)
    | SF SpecialForm


type SpecialForm
    = Define
    | Lambda
    | Display
    | If

I haven't done anything yet with Display and If.

The module Main is used to mediate via ports between the interpreter and the Javascript code that runs the command-line interface you saw above in the screenshot. This set-up is very small: just 43 lines of code for Main and 19 for repl.js.

Plans

When the project has matured sufficiently, I will publish a more fully-featured version of the code as an Elm library. I'm also planning to integrate the interpreter into Scripta.io, a web publishing platform for technical docs. Scripta offers three markup languages: MicroLaTeX, XMarkdown, and L0. The first two are cousins of their official counterparts, while L0 is an experimental effort with syntax inspired by Lisp.

Towards a Pure Elm Text Editor

James Carlson — Sat, 13 Feb 2021 17:45:37 +0000

I'd like to report on progress with a text-editor project I've been working on for some time. It is written purely in Elm, my favorite language for building web apps. I needed such an editor to build an IDE for another project, MiniLaTeX, and had found no other alternatives which suited me. But of course I hope that the editor, when finished, can be of use to others as well. Here is a demo and here is the GitHub repo.

The image below gives an idea of what the editor can do. Below the image I'll discuss features, challenges, and some of the technical details, as well as a bit of the story of how the project came to be.

The Story

First the story and some acknowledgements. I had been using the Elm equivalent of text areas for user input, but found a serious problem when the text was modified programmatically - the cursor would jump to the bottom! Very bad. I needed, for this and other reasons, a real editor. I tried some drop-in solutions but these did not work out well, so I started dreaming of a pure Elm editor that I could fiddle with at the lowest possible level. A big task, which I repeatedly put off.

One day, I chanced upon a prototype by Sydney Nemzer. While the demo this link points to might not seem like much, the code is well thought-out, really solid, and beautiful. To be honest, I had no idea how to solve many basic problems that Nemzer had already solved, e.g., how to interact with the cursor. Over the next two months, I grew Nemzer's code into a prototype that could do more and integrated it with the MiniLaTeX system. Sadly, as I worked on longer and longer LaTeX documents, the editor slowed down to a point that I was no longer happy with it.

About that time I heard a podcast with Martin Janiczek, where, among other things, he discussed what turned out to be an earlier prototype that Nemzer had based his work on. The difference: in Janiczek's version, the fundamental data structure was an array of lines, whereas in Nemzer's it was a single string. The latter is in many ways easier to work with, but the former is more efficient. Over the course of a couple of weeks, I rewrote the code to use an array. Lots of primitives working with text areas to replace basic string operations. Finally: much better performance!

Another challenge

Well, the long file story was not over. Web browsers slow to a crawl when the number of DOM nodes grows too large, and of course that would be the case if the editor loads the entire text of a large document into the DOM so that it can be scrolled in an unfettered way.

I decided to address the problem head-on by downloading a copy of Darwin's Origin of the Species, weighing in at more than 15,000 lines. This was to be my test file. The strategy would be to maintain a "window" into the full array of lines that was characterized by an offset and a "height," - the number of lines in the window. I've been using 600 lines for the window, with the editor displaying just 33 lines. We'll call the 33-line visible display the viewport. When the upper or lower edge of the viewport gets too close to the upper or lower edge of the window, the offset is repositioned so that the viewport is well inside it. This solution worked really well, and was quite snappy when editing The Origin of the Species.

Repositioning the viewport and window was slightly tricky - it should be unnoticeable to the user. Before solving that problem, of course, I had to redo the cursor-management code so as to take into account the offset.

On to new features!

This work done, I was finally ready to do what I had long wanted to do: implement LaTeX command completion. The idea is this. In LaTeX there are many constructions like

$$
\int x^n dx = \frac{1}{n+1}
$$

and

\begin{theorem}
There are infinitely many primes.
\end{theorem}

These are problematic in an app like minilatex.lamdera.app which updates the rendered LaTeX every 300 milliseconds or so. The reason is that if the user types \begin{theorem} without the closing \end{theorem}, the source text is in a particularly obnoxious error state that can extend to the end of the document. Ugh! One solution is have a keystroke sequence like ESC th ESC that inserts the text below, placing the cursor just before the XXX:

\begin{theorem}
XXX
\end{theorem}

This way, the user can be in a mostly error-free state at all times. (By "mostly", I mean avoiding really bad errors that extend over huge parts of the document because of unmatched begin-end pairs and the like).

The Developer Experience

In word, it was a good. Elm's type system helps me to think about what needs to be done and to translate ideas into code in a friction-free way. I've also found that it helps when I come back to a project after a long break, as I did several times with this one. In a very short time, I'm able to get the code "back into my head" and start forging ahead.

The fact that Elm is is compiled language is a huge advantage. A problem I faced in my previous life as a web developer was extreme FOR anxiety: Fear Of Refactoring: don't even think about the code, much less touch it: it might break! With Elm, by contrast, refactoring is a pleasure. Many times now I've changed a fundamental data structure deep in the core of the code, followed the compiler messages, and come out the other end an hour or two or three later with a fully functioning app - and the confidence that it is indeed free of lurking runtime bugs.

One last comment. Some years ago, I watched Rich Hickey's talk, Hammock-driven development. It is both entertaining and insightful. I used this methodology in the project several times with good results. Highly recommended!

Further technical notes

The app posted here was compiled using Matt Griffith's elm-optimize-level-2, then minified using uglify-js, resulting in a 307kb js file. This js file includes code for the editor, the MiniLaTeX-to-Html compiler, and a custom Markdown compiler, all written in Elm. Using elm make --optimize, the js file is slightly larger: 319kb.

The small asset size is due to the fact that elm make --optimize performs dead-code elimination on a per-function basis. If a module consists of one hundred functions, and only three (with their dependencies) are called by the code, then only those three survive compilation to Javascript.

I'll have to run some benchmarks on large LaTeX files to see to what extent elm-optimize-level-2 speeds things up.

Tooling

I've been using Umberto Pepato's Velociraptor for running all my development scripts ever since I learned about it on dev.to. Simple, light-weight, and perfect for the job. I love it! Velociraptor is written in Deno.

Making an Elm Desktop app with Tauri

James Carlson — Mon, 15 Jun 2020 09:40:40 +0000

I'd like to report briefly on my experience using Tauri to make a desktop version of an Elm app, provide it with access to a designated directory on disk, and package it for distribution.

In two words: awesome & amazing!

Setting up for dev

Time from starting to read the docs to bringing up the app with npx tauri dev was about 30 minutes. Had some very quick and accurate help on Discord from the dev team. I expect a certain amount of frustration and head-banging when trying something new. This was pleasant.

Running the app in dev

With one exception, the app worked as it had before running under Elm-live. The exception was some copy-paste functionality that uses ports and the navigator API. This gave me one of those fun undefined error messages in the console. (Right click on something in the app to bring up dev tools). I'll have to learn to use the Tauri API's.

Impressively, the other port-JS interop worked flawlessly, including (for me), the crucial MathJax.js library which renders LaTeX-style math formulas.

Bundling the app

A snap. I compiled and minified Main.elm as Main.js and copied it into ./dist. Ditto for index.html which references Main.js as well as one Css file and several JS files in ./assets. The directory ./assets was also copied into ./dist. With the files in place, I ran npx tauri bundle. After a while, the results were deposited as listed below:

$ tree -L 2 src-tauri/target/release/bundle/
src-tauri/target/release/bundle/
├── dmg
│   ├── app.dmg
│   ├── bundle_dmg.sh
│   └── support
└── osx
    └── muEdit.app

If you say “show package contents” for the app in osx, you find the file Contents/MacOs/app which weighs 4.9 MB. This is the executable.

Running the bundled app

Well, it just worked!: access to local and remote web servers, operation of the pure Elm text editor, compiling and rendering Markdown and (Mini)LaTeX, to HTML, rendering of math formulas, rendering of SVG images presented as strings, etc. And all quite snappy.

Next steps

I need to learn how to use the Tauri JS API to give access to the file system. The idea is for authors to be able to store all the work locally on disk as well as (if they wish) on a remote server. Folks like to have ownership of what they create. This may be a bit tricky: for Elm JS interop, I have to go through ports, referencing a file outside.js. But to get the file API, have to say const fs = require .... And it seems that that means some contortions with browserify. But it looks doable.

Conclusion

Tauri looks like a very promising way to bundle web apps. I'm looking forward to working with it more.

PS

If you have worked with both Elm and Tauri, please ping me here, on the Elm Slack or using gmail. Username is jxxcarlson everywhere. Would love to discuss this. Am also interested in Rust bindings.

PPS (Two days later)

It is now two days since starting my experiment with Tauri. I've implemented all the CRUD operations needed to persist documents to disk in a designated directory: create, list, read, update, and delete. These operations are carried out by one Elm module,Outside, that talks to one JS file, outside.js via a single pair of ports. (Murphie Randall's approach).

So yes, we can indeed use Tauri to build Elm desktop apps with the hard disk used to preserve state.

I would like to thank all of those who commented on an article I posted on Discourse on what turned out to be a poor initial strategy (using a localhost server). The comments on the post lead me to Tauri.

I would also like to thank the members of Tauri core team — @nothingismagick , @laegel , and @lucasfernog — who generously gave of their time to help solve several problems and points of confusion on my part. Their help made it possible to make fast progress.

Here is the Github repo

Why Elm is the right tool to build MiniLatex

James Carlson — Tue, 28 Aug 2018 01:48:44 +0000

LaTeX is the de facto markup language for writing technical articles with lots of mathematics in them — research articles, class notes, etc. In this post, I will describe how I used Elm to build a live-rendering web app for LaTeX. You can try out the demo at MiniLatex Live, which is pictured below. (This article continues below the image)

About LaTeX

A few words about LaTeX. The infrastructure for rendering it in print via PDF is excellent and well-established. For the web, there is a partial solution. A LaTeX document is made of two kinds of text — math mode and text mode. The formulas are in math mode, and the rest — section headings, tables of contents, cross references, theorem environments, macros, plain old prose, etc. — are in text mode. Mathjax or KaTeX can render the math mode material, but to my knowledge there has been to date no way of live rendering all of LaTeX. That is what MiniLatex, a 3k loc Elm library does. In a word, it uses a custom parser based on the elm/parser library to transform LaTeX source text into an abstract syntax tree (AST). Once one has an AST in hand, writing a renderer for the text mode material is fairly straightforward, with the math mode material delegated to MathJax.

The Beginning

I began working on the MiniLatex project just a little over a year ago, shortly after the Elm Europe conference in June 2017. That was when I realized that elm/parser may have the mojo to parse LaTeX. Happily, the parser lived in a framework for building Elm apps, so the combination was perfectly suited to my needs. Even now, a year later, the parser is only 300 lines of code in 3K loc library. But writing it was a real challenge for me: (a) I am a mathematician, not a computer scientist, and I knew only a little about parsing; (b) normally one begins with a grammar, then writes the parser — but there is no written grammar for LaTeX; (c) therefore, one has to reverse-engineer both a grammar and parser; (d) it turns out that any grammar for any adequate subset of LaTeX must be context-sensitive; (e) therefore the standared parser-generator tools cannot be used.

Obstacles

Despite these obstacles, of which I was (fortunately) for the most part only dimly aware, I began work on the parser. The elm/parser library is a parser-combinator library. It gives tools for building up a parser in a sensible way from little pieces. This is ideal for experimentation, which is what I had to do the absence of a grammar. The first step was to define an Elm type for the AST. Here it is:

type LatexExpression
    = LXString String
    | Comment String
    | Item LatexExpression
    | InlineMath String
    | DisplayMath String
    | SMacro String (List LatexExpression) (List LatexExpression) LatexExpression 
    | Macro String (List LatexExpression) (List LatexExpression) 
    | Environment String (List LatexExpression) LatexExpression
    | LatexList (List LatexExpression)
    | LXError (List DeadEnd)

Just 11 lines of code that governs a 300 line parser which is the heart of a 3000 line library. I did not write all 11 lines in one go. I wrote a few lines, tried to build a parser that would handle them, and then iterated.

Slow going at first

The going was quite slow at first. Evan Czaplicki had brought elm/parser, parser combinator library. Although I had read a little about parser combinators, I had never used them. Thankfully, I had much generous assistance from community on the Elm Slack, and Ilias van Peer, who hangs out there frequently, helped me over many, many rough spots. (Thankyou Ilias!) Bye and bye I got the hang of it.

Over the next few months, things came together, and by the time of the of the Elm Conf in September of 2017, I had a decent working prototype. But there was a problem. The parser was too slow! I could easily take 15 seconds to parse a few pages of text. I was chatting back and forth with Ilias about this, who discovered the bottleneck was — characters were boxed into a JS object (as I understand it). The problem was likely to be fixed in the 0.19 version of the compiler, so in the meantime, using another of Ilias' suggestions, implementing a diffing strategy that made incremental parse-render operations (after the initial, full-document parse) very fast.

0.19!!!

All of these efforts came together when the alpha of 0.19 came out. The elm/parser library had been rewritten and as a result the MiniLatex parser (which also had to be rewritten) was incredibly fast. That speedup, combined with the diffing strategy, made live-rendering of LaTeX documents a reality. I hadn't thought that this would be possible, but yes, it was!

Comments

Elm was my first experience with a typed functional language. It took me a while to understand the type system well, and also to appreciate its power. I now realize that this is one of Elm's great strengths. Of course, we as developers experience it in our daily work when we embark on an extreme refactoring operation and come out of the battle an hour or two later with everything working. (Thank you, compiler, for your helpful messages!) But in this project, the type system played a central role. The real beginning of MiniLatex was the moment when I wrote down a first version of the LatexExpression type. The code for the parser grew out of this type definition, and it has played a guiding role throughout.

The success of the MiniLatex project was made possible by the generous help of the community on the Elm Slack, and by Ilias in particular. I would also like to thank Evan Czaplicki and Luke Westby, who helped me at some critical points in the transition from 0.18 to 0.19. It has been a joy to work with Elm, and I would like to give a big shout out of appreciation to Evan and the core Team. Bravo! What a fantastic thing you have made!!

Caveat

MiniLatex is two things: a proper subset of LaTeX and a parser-render for that subset. The subset, which is of course subject to expansion as time and energy permit, is nonetheless adequate to write some pretty heavy lecture notes. MiniLatex documents can also be exported and run through standard machines like pdflatex.

Note

MiniLatex is embedded in an app at knode.io which provides a live editing environment, a searchable repository where the documents you write a are stored, an image uploader, and a searchable image catalogue.