DEV Community: Joe Nash

Enso Dev Blog - 19th June 2020

Joe Nash — Fri, 26 Jun 2020 12:18:30 +0000

Welcome to the inaugural Enso devblog. In this series, we will share the latest updates from the development of Enso (formerly Luna), a visual language for data processing.

Going forward, we’ll be sharing these updates at the end of each sprint, so you can expect an update every two weeks. This update is from the sprint ending on the 19th June 2020. If you want to keep up with the development of Enso in real time, you can follow along on GitHub.

A new name for a new codebase.

Luna is now Enso. Following a couple of years of going by Luna, we were facing issues that were making it difficult for us, and people looking for Luna. Luna is a popular term, and in programming-language land, is also very close to the popular language Lua, an endless source of confusion.

Our new name, Enso, is inspired by the Japanese 円相 (en-sō), “a circle that is hand-drawn in one or two uninhibited brushstrokes to express a moment when the mind is free to let the body create”. This concept elegantly captures what we are trying to achieve with our visual language, and is a guiding principle in ensuring that Enso is as intuitive, and natural to use, as possible.

You can find us at our new home, https://enso.org. Whilst we continue rebranding, you may still see references to Luna here and there.

As well as a new name, Enso has a completely new codebase. Luna 1.x, culminating in Luna 1.9, was a fantastic proof that what we set out to do is achievable. We were able to quickly validate the design of our dual representation visual language (where both the textual and visual versions of the workflow are equivalent and simultaneously updated), and confirm that many of the features we wanted to see in such a language, such as immutable data structures and currying, worked in the visual medium.

Thanks to the participation of our community in using and contributing to 1.x, we were able to gather a huge amount of feedback, and points of improvement, including in areas of core language design. For example, the feedback from users of 1.x lead us to recognise the necessity of a feature we now call Uniform Call Syntax, which you can learn more about here.

So why start from scratch, and not continue to iterate on 1.9? The design decisions that allowed us to make those quick iterations proved to be detrimental to future progress. For example, the GUI was slow, based on SVGs, and integrated with the Atom editor. It would be difficult, if not impossible, to get the performance we wanted building on that foundation.

Enso is a complete rewrite, taking that validated design, and now focusing on performance and usability. Firstly, the new JIT compiler is built on GraalVM, a highly-performant Java VM. We are already seeing performance in some cases of 2 orders of magnitude faster than Python. The compilation process itself is also very fast, giving immediate feedback, particularly important in a visual language.

The performance of the GUI is night and day compared to Luna 1.x, rewritten from scratch with Rust, and compiled to WebAssembly. We’ve moved away from SVGs in Atom, to our own WebGL engine, heavily optimised to display 2D graphics, allowing for graphs built of thousands of nodes to run in 60 FPS. This engine will also power our new visualisations, displaying millions of points at 60 FPS.

Thanks to GraalVM, we are also able to include some heavily demanded features. Enso will have a much bigger potential for foreign language interoperability — first with Java, but soon Python and R, bringing your existing data processing toolkit to Enso.

We hope to publish a full article reflecting on our previous tech stack decisions, and why we’ve made the changes we have. But for now, let us show you what we’ve been working on.

Can I use Enso?

Enso is currently pre-1.0, and is not ready for day to day data processing use cases. Most notably, there is currently no standard library. For now, intrepid adventurers can try out the latest builds via the artifacts in the GitHub repositories. You can find the instructions for getting and running Enso here.

Integrated Development Environment

The Enso IDE will include everything an end-user needs to process data with Enso. The IDE will support a user in writing both visual and textual Enso, with the tools and convenience expected from a modern IDE, such as auto-complete and context sensitive search.

Nodes

Users of Luna will notice that we’ve adopted a new node shape in Enso. Previously in Luna, the expression that defined a node would be displayed above the circular node, with the ports arranged along the outside of that circle. Now the node is the expression, and the input ports are the parameters of the expression.

As Enso workflows are built from connecting these nodes, we’ve been putting a lot of work into making those connections as seamless, and pleasant as possible.

Here you can see some of the new node and edge interactions:

Creating edges from input nodes.

Disconnecting edges, with highlighting for the edge to be disconnected.

Support for many (MANY) output ports.

There is also some additional handling to prevent triggering a node interaction as you mouse around your canvas, say from one node to another. You may pass your mouse over another node. On a quick cursor over, the ports do not appear, requiring a longer interaction:

As you add more nodes to your canvas, you’re going to want to move them around and organise them. Snapping makes that a breeze, helping you align graphs in your workflow:

Text editing

Enso is dual representation, meaning the visual representation (nodes) are equivalent with a more traditional, textual programming language. Enso needs a text editor as beautiful and usable as its nodes.

This sprint, the Enso text editor got rich text display capabilities. It now supports colours per glyph, formatting such as bold, italics and strikethroughs, and widgets, such as sliders and cooler choosers.

Coloured text for future syntax highlighting.

The rendering engine has been integrated with the Xi Editor rope-based backend. Thanks to Raph Levien and Trisan Hume from the Xi Editor team for all of their support and help!

Dual representation

Numerous bugs in the module responsible for keeping the textual and visual representations in sync have been fixed, and support for runtime type information has been added. This change is not yet visible, but is a precursor to enabling support for types in the visual representation. Users of Luna may recall that ports and connections between nodes were coloured by the type of data they acted upon, which will now be possible in Enso.

Dual representation correctly discovers nodes and connections, and updates the textual representation in real time as a user edits the node:

Real-time updating between visual and textual representation.

Project management

Support for projects was added to the IDE, and it is now possible to open or create projects when invoking the IDE via the --project argument:

Invoking Enso Studio (the IDE) with the --project argument to open existing projects.

Engine

The Enso Engine includes the compiler, runtime and language server, and will, in the future, include the type checker. It is used as the backend of the Enso IDE, but can also be used in isolation.

Language Server

This sprint saw the start of support for auto-completion in the IDE added to the language server.

Auto-complete suggestions are obtained by querying the Semantic Database. The database is updated by extracting autocomplete entries from the compiled Enso workflow. The IDE holds its own view of the Semantic Database, by retrieving it from the server and subscribing for subsequent updates.

Language improvements

Support for the type signature syntax was added. Whilst the signatures are not yet checked, initial support for suspended arguments has been added.

Support for suspended arguments in type signatures.

Booleans have been added to the language. These so-called “magic booleans” look like a normal datatype, but are actually well-optimised primitives under the hood.

Check out these examples:

Runtime

Support for running “slow-path” Enso. As a functional language, Enso attempts to perform a variety of optimisations for function calls to ensure they are efficient, and easily inlined. However, there are cases, such as interpreter entry points and very polymorphic call sites, where caching and inlining would not be possible, taking too much time and memory. In this case, Enso takes the “slow-path”, which would be faster in those cases than attempting caching and inlining. You can learn more about “Megamorphic Call Sites” in the runtime docs.

More information

That’s all for this developer update. We’ll be back with more after the next sprint. You can continue to follow along with Enso development on GitHub, by joining our Discord server, or subscribing to updates on our developer mailing list.

Deploying to Heroku from GitHub Actions

Joe Nash — Tue, 07 Jan 2020 22:12:20 +0000

GitHub Actions give us a new way to deploy to Heroku, and to integrate Heroku with other parts of our development workflows. In a single GitHub Actions workflow, we could lint our Dockerfile and package configs, build and test the package on a variety of environments, compile release notes, and publish our app to Heroku.

Today we are going to build a simple workflow that will use the Heroku CLI to deploy a project with a Dockerfile to Heroku via the Heroku Container Registry. During the course of building this workflow, we will see how to create jobs, configure the runner environment with GitHub secret store, use public actions, and react to and filter events.

What is GitHub Actions?

GitHub Actions are a way to trigger custom workflows in response to events on GitHub. For example, a push could trigger Continuous Integration (CI), a new issue being opened could trigger a response from a bot, or a pull request being merged could trigger a deployment.

Workflows, jobs, and actions

Workflows are made up of jobs. These jobs are performed by a runner in a virtual environment. Jobs are composed of individual steps, and those steps can be scripts executed on the runner, or an action. Actions are composable units of workflows, that can be built in Docker containers or JavaScript, placed directly in your repository, or included via the GitHub Marketplace or Docker registry.

Getting started

To get started, push a project with a Dockerfile to GitHub, or fork an existing repository. Don’t forget that Heroku provides an environment variable, $PORT, for apps to bind to for HTTP traffic, so be sure to adjust the project to listen to that variable, rather than an explicitly set port.
Within the repository, navigate to the Actions tab.

On first opening the Actions tab, we’ll be presented with some options for getting started. These include CI setups for popular languages, as well as a simple example workflow. Let’s start by selecting that simple workflow. Rename the file to workflow.yml, and hit “Start commit”.

Your first workflow

After the workflow is setup, the repository will have a .github/workflows folder, with a workflow.yml file inside.

This workflow defines one job, build, with 3 steps. One of those steps uses an existing, public action, checkout. The others are defined within the job, with a name, and a run field. run executes commands on the runner performing the job. This job will use an Ubuntu machine. All GitHub Actions environments have the same specs, but you can run jobs on Ubuntu, MacOS, and Windows.

checkout is an important action that you will use in most workflows that work on the code in the repository. checkout fetches the contents of your repository to $GITHUB_WORKSPACE, an environment variable that maps to /home/runner/work on the runner. checkout, like many first-party actions provided by GitHub, is open source and viewable on the Actions GitHub Organization.

This workflow runs on the push event, that is any time new contents are pushed to the repository. You can see the workflow and the status of its run under the Actions tab.

Inside the workflow, you can see that each of the named steps has logs that can be expanded. Naming and describing the steps in your workflows can help this interface provide rich information on the state of your deployments.

Whilst the repository contents are fetched to the runner with checkout, the workflow does not yet make any use of them. Let’s start working towards having this Dockerised project published to a Heroku app.

Environment variables and GitHub Secret store

You can deploy to the Heroku Container Registry with either the Heroku CLI, or Docker. Fortunately, both are available within the virtual environment provided to the runners, when using Ubuntu. Using the Heroku CLI will give us access to other useful utilities, so let’s start there.

Browserless Heroku CLI Authorization

In order to build or deploy, you will have to login to the Container Registry. When using the Heroku CLI locally, login is via the browser, but the Heroku CLI can also be authenticated by providing an OAuth token.

To create an OAuth authorisation, on your local machine, run:
heroku authorizations:create.

This will create a long-lived user authorization, whose token can be used to authenticate the Heroku CLI in our workflow.

The Heroku CLI expects this token to be found in an environment variable, HEROKU_API_KEY. You can define environment variables within job steps, but you don’t want to insert the key directly and commit it to the repository. Luckily, GitHub Actions comes with a new Secrets store, within the repository settings.

There is a prompt to add a new secret.

Within Secrets, create a new secret, HEROKU_API_KEY, and insert the token given by the Heroku CLI.

Going back to our workflow.yml, add a new step named “Login to Heroku Container Registry”. Within this step you need to define the environment variable HEROKU_API_KEY by grabbing the secret, followed by running container:login with the Heroku CLI.

Workflow environment variables

Environment variables are defined with env. The workflow can access the contents of the repository secret store through a Context. There are many Contexts available which hold information about the workflow run, the job being performed, the runner environment, and the secret store. Retrieve the secret from the secrets Context and assign it to an environment variable like so:

With that environment variable available, you can now run heroku container:login to log into the Heroku Container Registry with the OAuth token.

Commit that code and push it to the repository to trigger the workflow.

Back under the Actions tab, your step will execute and log you in successfully:

Build and Release to Heroku Container Registry

Now that the Heroku CLI is authenticated against the Heroku Container Registry, you can push your project to be built, and then release the resulting container.

Create two new steps, one for the push, and the other for release. Each will also need a declaration of the HEROKU_API_KEY environment variable, as environment variables are not persisted between steps.

When pushing and releasing the container, you can specify an app name to target. This could be included safely in the workflow .yml, but as the app name is used in multiple commands, and you may want to change it later, let’s add it to the secret store and access it via the secrets context.

Commit and push those changes, and return to the GitHub repository Actions tab to check on the build:

The build has completed successfully, and you should now be able to visit your deployed app on Heroku.

Events and Filters

Right now, the push event is triggering this workflow, regardless of branch, or file, that new code is pushed to. You will likely want to tailor this trigger depending on your development practice. For example, if using the GitHub Flow, you may want to deploy to a staging environment when a pull request is merged to master.

Workflow syntax gives us the ability to filter on branches, and files, as well as to trigger on all of the available GitHub webhook events. Let’s modify our push event to filter only for pushes to master.

Commit that, and push to master. Now any subsequent commits to branches other than master will not trigger this workflow.

Summary

You’ve now created a GitHub Actions workflow, that:

On a push to master
Fetches the contents of the repository
Logs into Heroku Container Registry
Builds the container on Heroku
Publishes the project to an Heroku app

What’s next?

From this workflow, you can add new steps, and jobs, for other tasks in your development process. For example, why not use the Docker Lint action to lint the Dockerfile, before pushing to Heroku Container Registry?

There are also other ways to implement Heroku in a GitHub Actions workflow. This example used the Heroku Container Registry, via the Heroku CLI already installed on the runner virtual environment. The virtual environments also come with Git, so with minimal modification, you could use this workflow to deploy projects without a Dockerfile.