DEV Community: Sebastian Riedel

mojo.js 1.0 - from Perl to Node.js

Sebastian Riedel — Mon, 20 Jun 2022 14:57:14 +0000

After one year of development work we are happy to finally announce the very first major release of the mojo.js web framework for Node.js. Now available on GitHub and NPM.

Here's the obligatory "hello world" single file app, with WebSockets:

But mojo.js is not really about single file apps. As a very traditional hypermedia framework and spiritual successor to Mojolicious, it strongly encourages a Model-View-Controller (MVC) layout, while also supporting these single file apps for prototyping.

But, Why?

Right now almost all JavaScript web frameworks are split into two categories. On one side you have middleware frameworks, that do pretty much nothing else than routing on their own, so with every new project you have to build your own framework from scratch with dozens of middleware layers. And on the other side you have the kitchen-sink, where the framework makes every decision for you, forcing you to use React on the frontend, or MongoDB as the database, often even limiting your hosting options to a few large cloud providers. Especially the latter kind of web framework tends not to age very well.

We believe that there is still ample room in the middle for mojo.js. A framework that provides just the essential building blocks for backend web services. The things that rarely change from project to project. Like routing, serving static files, server-side rendering, logging, config files, form validation... you get the idea. In ten years from now you should still feel confident relying on them.

Aside from reliability, having components specifically designed to be used together allows for significant performance optimisations. That's why mojo.js is a lot faster than Express and Koa for example, despite having many more features.

From Perl to Node.js

The Mojolicious project is a group of polyglot programmers who started out with Perl back in the hay days of CGI scripting. Some of us have been making mainstream web frameworks for two decades now. From Catalyst in 2004 to Mojolicious in 2010. Powering some of the largest sites on the web along the way.

Ever since Perl6 (now Raku) started to become a thing we've had plans to port Mojolicious to more languages than just Perl5. But Perl6 drifted into a different direction than what we were hoping for and so those plans never truly materialised.

However, at the same time JavaScript kept evolving. The language gained features like ES6 classes, async/await, ES modules, arrow functions, const/let keywords and much more. Node.js finally brought JavaScript to the server-side. On the language level, there's a pretty close relationship between Perl and JavaScript (and it's not just sharing use strict or having native Regex data types). So it was inevitable that some of us would grow to like JavaScript quite a bit.

Not quite full-stack web framework

I still remember when full-stack meant that the framework contained a router, template engine, and an ORM with support for a bunch of SQL databases. These days they include things like a custom React distribution and a subscription for a serverless hosting service. It's hard to escape these ecosystems again without changing your whole tech stack.

With mojo.js we don't do most of those things. While it ships with a router and server-side renderer with support for multiple template engines, there is no default database. Just a workflow for adding your own model layer. Similarly choosing a frontend framework is entirely up to you. The static file server will deliver whatever assets you need.

What's most important for us, is to provide you with a rock solid foundation. RESTful routing, WebSockets, static file server, cli, logging, config files, session management, form and JSON validation, content negotiation, TypeScript types and a testing framework. Components you can rely on for decades if necessary.

Software supply chain attacks around NPM are a hot topic right now. And it can be quite problematic to audit your whole dependency tree. Thankfully, while the Perl version has no dependencies at all, in JavaScript we only have 23 trusted third party dependencies. And we are prepared to replace every single one of them if necessary, with a port of the battle-tested Perl implementation.

We've also started releasing spin-off projects for standalone use outside of mojo.js, based on some of our other popular Perl projects. @mojojs/dom, @mojojs/template and @mojojs/path follow the same strict rules as mojo.js itself.

What about Mojolicious?

Now you might be wondering if we are going to abandon Mojolicious. And the answer is a resounding no. All of us still enjoy Perl very much, and look forward to keeping it alive for at least a few more decades. 😉

Have fun!

Playwright and Mojolicious

Sebastian Riedel — Fri, 26 Mar 2021 17:08:37 +0000

It's Hack Week again at SUSE. 🥳 An annual tradition where we all work on passion projects for a whole week. Some of us make music, others use the time to experiment with the latest technologies and start new Open Source projects.

My project this time was to see if there's a better way for us to do automated browser testing of our Mojolicious web applications. For a long time Selenium has been the de facto standard for browser automation, but these days there are more modern alternatives available, such as Playwright. But how good is Playwright really? Spoiler: It's very good. But keep reading to find out why and at what cost.

What is Playwright?

Playwright, just like Selenium before it, is a framework for browser automation. You can use it for all sorts of scripted interactions with websites, such as buying those Nvidia GPUs from online retailers faster than everyone else 😉, but it is most commonly used in test suites of web applications.

The code is being developed by Microsoft as an Open Source project with Apache 2.0 license and distributed as an NPM package. So all you need is Node, and you can install it with a one-liner.

$ npm i playwright

There are bindings for other languages, but to get the most out of Playwright you do want to be using JavaScript. Now when it comes to browser support, where Selenium would give you the choice to pick any WebDriver compatible browser as backend, Playwright will download custom builds of Chromium, Firefox and WebKit for you. And that's all you get.

They are doing it for pretty good reasons though. The browser binaries tend to work flawlessly on all supported platforms, which currently include Windows, macOS and Linux (x86). And when you are used to the sluggishness of Selenium, it almost seems magical how fast and reliable Playwright runs.

This is because where Selenium sticks to open protocols, Playwright will use every trick in the book for better performance. Including custom patches for those browsers, extending their DevTools protocols, and then using those protocols to control the browsers. I'm not a huge fan of the approach, but it's hard to argue with the results.

Short term there are huge benefits, but having to maintain these browser patches indefinitely, if they don't get merged upstream, might hamper the longevity of the project.

Using Playwright

import assert from 'assert/strict';
import { chromium } from 'playwright';

(async () => {
  const browser = await chromium.launch({ headless: false, slowMo: 50 });
  const context = await browser.newContext();
  const page = await context.newPage();

  await page.goto('https://mojolicious.org/');
  await page.click('text=Documentation');
  await page.click('text=Tutorial');
  assert.equal(page.url(), 'https://docs.mojolicious.org/Mojolicious/Guides/Tutorial');
  await page.screenshot({ path: 'tutorial.png' });

  await context.close();
  await browser.close();
})();

If you've ever done web development before, the API will be very intuitive, and it was clearly designed with async/await in mind, which i'm a huge fan of. You can have multiple isolated browser contexts, with their own cookies etc., and each context can have multiple pages.

Every interaction, such as page.click(), will automatically wait for the element to become visible, with a timeout that defaults to 30 seconds. This is a huge step up from Selenium, where you have to build this logic yourself, and will get it wrong in many many entertaining ways. 😅

You can emulate devices such as iPhones, use geolocation, change timezones, choose between headless and headful mode for all browsers, and have the option to take screenshots or make video recordings at any time.

One of the latest features to be added was the GUI recorder, which opens a Chromium window, and then records all user interactions while generating JavaScript code as you go. I was a bit sceptical about this at first, but it can significantly speed up test development, since you don't have to think too much about CSS selectors anymore. Even if you just end up using parts of the generated code.

Playwright and Perl

Running Playwright against live websites is very straight forward. But for automated testing of web applications you also want your test scripts to start and stop the web server for you. And this is where things get a little bit tricky if your web application happens to be written in a language other than JavaScript.

use Mojolicious::Lite -signatures;

get '/' => {template => 'index'};

app->start;
__DATA__
@@ index.html.ep
<!DOCTYPE html>
<html>
  <body>Hello World!</body>
</html>

What i needed to run my Perl app was a JavaScript superdaemon with support for socket activation. Unfortunately i've not been able to find a module for the job on NPM, and had to resort to writing my own. And now the Mojolicious organisation is not just on CPAN, but also on NPM. 😇

import assert from 'assert/strict';
import ServerStarter from '@mojolicious/server-starter';
import { chromium } from 'playwright';

(async () => {
  const server = await ServerStarter.newServer();
  await server.launch('perl', ['test.pl', 'daemon', '-l', 'http://*?fd=3']);
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  const url = server.url();

  await page.goto(url);
  const body = await page.innerText('body');
  assert.equal(body, 'Hello World!');

  await context.close();
  await browser.close();
  await server.close();
})();

You might have noticed the odd listen location http://*?fd=3. That's a Mojolicious feature we've originally developed for systemd deployment with .socket files. The superdaemon, in that case systemd, would bind the listen socket very early during system startup, and then pass it to the service the .socket file belongs to as file descriptor 3. This has many advantages, such as services being started as unprivileged users able to use privileged ports.

Anyway, our use case here is slightly different, but the same mechanism can be used. And by having the superdaemon activate the socket we can avoid multiple race conditions. The socket will be active before the web application process has even been spawned, meaning that page.goto() can never get a connection refused error. Instead it will just be waiting for its connection to be accepted. And important for very large scale testing, with many tests running in parallel on the same machine, we can use random ports assigned to us by the operating system. Avoiding the possibility of conflicts as a result of bad timing.

Combining Everything

And for my final trick i will be using the excellent Node-Tap, allowing our JavaScript tests to use the Test Anything Protocol, which happens to be the standard used in the Perl world for testing.

#!/usr/bin/env node
import t from 'tap';
import ServerStarter from '@mojolicious/server-starter';
import { chromium } from 'playwright';

t.test('Test the Hello World app', async t => {
  const server = await ServerStarter.newServer();
  await server.launch('perl', ['test.pl', 'daemon', '-l', 'http://*?fd=3']);
  const browser = await chromium.launch();
  const context = await browser.newContext();
  const page = await context.newPage();
  const url = server.url();

  await page.goto(url);
  const body = await page.innerText('body');
  t.equal(body, 'Hello World!');

  await context.close();
  await browser.close();
  await server.close();
});

You might have noticed the shebang line #!/usr/bin/env node. That's another little Perl trick. When the Perl interpreter encounters a shebang line that's not perl it will re-exec the script. In this case with node, and as a side effect we can use standard Perl testing tools like prove to run our JavaScript tests right next to normal Perl tests.

$ prove t/*.t t/*.js
t/just_a_perl_test.t ... ok                                                                                                     
t/test.js .. ok
All tests successful.
Files=3, Tests=4,  2 wallclock secs ( 0.03 usr  0.01 sys +  2.42 cusr  0.62 csys =  3.08 CPU)
Result: PASS

In fact, you could even run multiple of these tests in parallel with prove -j 9 t/*.js to scale up effortlessly. Playwright can handle parallel runs and will perform incredibly well in headless mode.

One More Thing

And if you've made it this far i've got one more thing for you. In the mojo-playwright repo on GitHub you can find a WebSocket chat application and mixed JavaScript/Perl tests that you can use for experimenting. It also contains solutions for how to set up test fixtures with wrapper scripts and how to run them in GitHub Actions. Have fun!

High Priority Fast Lane for the Minion Job Queue

Sebastian Riedel — Sat, 06 Mar 2021 16:10:59 +0000

Minion is a high performance job queue designed for use with Mojolicious. In this article i will introduce you to one of the new features in the 10.17 release.

The Problem

Recently in one of my work projects at SUSE i ran into a job queue congestion issue. We have a lot of very slow background jobs to perform various maintenance tasks, such as cleaning up old files from disk that are no longer needed. Some of these jobs can take over an hour to finish and they are not particularly time critical, so very low priority.

Now, due to some bad luck we had a large backlog of these one hour jobs. And at some point there were no higher priority jobs left in the queue. So the Minion worker ended up processing only one hour jobs at full capacity.

This wouldn't have been a problem if we didn't also have time critical high priority jobs come in right at that moment. With no extra capacity left, they had to wait the full hour. And since those high priority jobs were created by users of the web service, they were of course assuming something was broken when they didn't receive a result in a timely manner. And we ended up with a whole bunch of bug reports.

A Fast Lane

Of course this is not a particularly uncommon problem. Larger web services would simply use multiple Minion workers with different named queues for various service levels. Probably with different hardware for paying customers and those using a free plan. But this is a smaller service that some users also install on their own machines locally. So one Minion worker is more than enough under normal circumstances.

What we needed was a simpler solution. The prefork web server that ships with Mojolicious has this concept of spare processes. Where under heavy load it is allowed to spawn a few extra processes for request handling at peak times. While normally it would only keep a smaller number of processes preforked.

The same concept can also be applied to the Minion worker. We simply set aside a small number of jobs from our maximum capacity only to be used for high priority jobs. So instead of just a --jobs 4 setting, we now also have a --spare 2 setting. And at peak times there will be 6 jobs running in parallel, of which at least 2 are guaranteed to be high priority. What exactly is considered high priority can be controlled with the --spare-min-priority 5 setting, which will default to 1, since the default priority for all jobs is 0 and lower priority jobs are usually given negative priority numbers.

$ script/myapp minion worker --jobs 4 --spare 2
[2021-03-06 13:50:22.02638] [30272] [info] Worker 30272 started
...

The Implementation

Since Minion relies on PostrgeSQL the actual implementation was very simple. Just two lines of Perl to spawn spare processes and a one line change in the SQL query used to dequeue jobs.

UPDATE minion_jobs SET started = NOW(), state = 'active', worker = ?
WHERE id = (
  SELECT id FROM minion_jobs AS j
  WHERE delayed <= NOW() AND id = COALESCE(?, id) AND (parents = '{}' OR NOT EXISTS (
    SELECT 1 FROM minion_jobs WHERE id = ANY (j.parents) AND (
      state = 'active' OR (state = 'failed' AND NOT j.lax)
      OR (state = 'inactive' AND (expires IS NULL OR expires >
-  )) AND queue = ANY (?) AND state = 'inactive' AND task = ANY (?) AND (EXPIRES IS NULL OR expires > NOW())
+  )) AND priority >= COALESCE(?, priority) AND queue = ANY (?) AND state = 'inactive' AND task = ANY (?)
+    AND (EXPIRES IS NULL OR expires > NOW())
  ORDER BY priority DESC, id
  LIMIT 1
  FOR UPDATE SKIP LOCKED
)
RETURNING id, args, retries, task;

Luckily the index we use for the ORDER BY also works for the minimum priority check, so no further optimisations were needed. We can still dequeue thousands of jobs per second. I love PostgreSQL.