DEV Community: Doug Bell

A New Type of Mojolicious Frontend

Doug Bell — Sun, 21 Nov 2021 23:02:19 +0000

JavaScript has evolved quite a bit from its humble beginnings. Unfortunately, some of those advances have been difficult to take advantage of. Some require features not supported by all browsers, others require complicated tools. For my latest Mojolicious web app, I wanted to see how I could use Typescript with a minimum of fuss.

We'll start from a simple Mojolicious::Lite app, created using mojo generate lite_app:

From there, we can install Typescript using Node. Since Typescript cannot bundle itself into a single file, we will also install Webpack:

npm install --save-dev typescript webpack webpack-cli ts-loader

After installation we can configure Typescript using its tsconfig.json file. We'll use the public directory for our output, since this directory is served by Mojolicious.

{
  "compilerOptions": {
    "outDir": "./public/",
    "sourceMap": true,
    "noImplicitAny": true,
    "module": "es6",
    "target": "es5",
    "jsx": "react",
    "allowJs": true,
    "moduleResolution": "node"
  }
}

We'll also need to configure Webpack's webpack.config.js file to get it to bundle everything into a single app.js file we can use. We'll have it start looking for our app in src/index.ts and write our bundled app to public/app.js.

const path = require('path');

module.exports = {
  entry: './src/index.ts',
  devtool: 'inline-source-map',
  module: {
    rules: [
      {
        test: /\.ts$/,
        use: 'ts-loader',
        exclude: /node_modules/,
      },
    ],
  },
  resolve: {
    extensions: [ '.ts', '.js' ],
  },
  output: {
    filename: 'app.js',
    path: path.resolve(__dirname, 'public'),
  },
};

With this complete, we can create a basic Typescript program.

// src/index.ts
function component() {
  const element = document.createElement('div');
  element.innerHTML = ['Hello,', 'Mojolicious!'].join(' ');
  return element;
}
document.body.appendChild(component());

We can then build our Typescript program using Webpack:

$ npx webpack
asset app.js 864 bytes [compared for emit] [minimized] (name: main)
./src/index.ts 196 bytes [built] [code generated]

webpack 5.64.2 compiled in 3123 ms

This creates public/app.js, which we can then add to our Mojolicious app. To automatically rebuild our app.js when needed, we can run npx webpack --watch.

For a complete example, see this sample repository for a Mojolicious/Typescript app. As a bonus, you can get full end-to-end testing for your Mojolicious and Typescript web app using Playwright. With all this, you can bring the best that modern JavaScript has to offer to your Mojolicious web apps!

Yancy: What We Leave Behind

Doug Bell — Thu, 04 Nov 2021 01:06:11 +0000

When I started the Yancy Content Management System, my goal was to see how easy it would be to build a generic admin editor on top of a Mojolicious application database. Its design was, therefore, simple: A backend layer to talk to the database, a web application to view and edit the data, and an API controller to connect the two.

Now, 4 years later, Yancy is on its way to becoming the M in Mojolicious MVC. I added Yancy::Model as a place to keep data logic without the overhead of DBIx::Class, but having a layer between the database backend and the web API enables all sorts of fun things, including:

Custom validation
Packing and unpacking of complex data
Automatic joining of related data
Secondary, read-only backends and failover
Transparent caching
Data versioning

And these things can work across different backends: A caching module that uses Redis could be a frontend for a cache of a MySQL database or a Postgres database. The code only needs to be written once and made available on CPAN.

Unfortunately, the new APIs take the place of some existing APIs and features that will have to be removed. These features were added to serve the frontend, but are awkward and limited in power compared to customizing the model API. The main features that have been deprecated are: Filters, Views, and OpenAPI.

Filters were added to enable the frontend to do password hashing. By adding x-filter to the configuration, a field or a row could have one or more filters applied to it before being written to the database. With the addition of the model API, filters can be added more easily and explicitly by creating a custom Schema class.

Views were a way to present subsets of data to a user through the frontend application. Instead of seeing all the raw information an application needs for a user, the editor could be made to show just the important information for the administrator or content manager. This, much like filters, can be done more robustly through the model API, or even directly through your database.

Last, the OpenAPI spec was how the frontend app analyzed the data schema to determine what it could do. However, since every schema can only do the same four operations (create, read, update, and delete), reading the OpenAPI spec to handle operations is needless complexity. The generated API could be used by more than just the Yancy editor, so the code for generating OpenAPI specs from Yancy schemas has been moved to the Yancy::Plugin::OpenAPI module on CPAN.

Yancy v2 has been planned and in development for quite a while, but it is nearing completion. When v2 is released, these deprecated features will be removed. What will remain is a leaner, easier-to-use content management system for the best web framework out there.

Yancy: The Next Model

Doug Bell — Mon, 06 Sep 2021 18:15:36 +0000

Yancy is a content management system and application framework for the Mojolicious web framework. For the last year, I've been using it to develop Zapp, my workflow automation webapp. Over that time, it became harder and harder to organize all the code needed to manipulate Zapp's data. To solve this problem, I wrote Yancy::Model.

Mojolicious is a Model-View-Controller (MVC) web framework. The model layer is where the data manipulation happens: Reading and writing records in the database. It is also where business logic happens: Sending e-mail for transactions, or periodic data cleanup. The goal of a model layer is to provide an API on to the application's data so that it can be used not only by the web application, but by other tools as well.

Because Mojolicious does not provide its own model layer, most people usually turn to the DBIx::Class ORM. But, this has always felt too complex and heavy for my needs: By the time I know my project needs something like DBIx::Class, it's usually too late to migrate easily. I wanted something lighter and more agile that could grow from a rapidly-developed proof-of-concept app to the final, maintainable production version.

Yancy provides a generic API on to multiple database systems, which it calls Backends. Backends handle basic database operations with a common API. Using this common API, I built a lightweight class system for accessing data:

Yancy::Model wraps the backend object and manages the classes.
Yancy::Model::Schema provides methods to create and search a database table.
Yancy::Model::Item represents a single row in a table and provides methods to update and delete it.

Using these basic classes provides a more fluent interface to the database than using the backend API directly. But, the power of a model layer is in writing custom code to make managing the data easy and safe. For this, Yancy::Model allows you to add your own classes for schemas (tables) and items (rows).

Writing custom model classes makes organizing your data management code easy. For example, Zapp has a table for workflows (called "plans") with a related table containing tasks ("plan_tasks") and another related table containing workflow input ("plan_inputs"). Whenever a user looks at a plan, they need to also see the plan's tasks and inputs. So, I can create a schema class that fetches this information automatically.

package Zapp::Schema::Plans;
use Mojo::Base 'Yancy::Model::Schema', -signatures;
sub get( $self, $id, %opt ) {
    # I could use two JOINs here instead, but joining 
    # two 1:* relationships could result in a lot
    # of data to fetch and discard...
    my $plan = $self->SUPER::get( $id, %opt );
    my $inputs_schema = $self->model->schema( "plan_inputs" );
    $plan->{inputs} = $inputs_schema->list({ plan_id => $id }, { order_by => 'rank' })->{items};
    my $tasks_schema = $self->model->schema( "plan_tasks" );
    $plan->{tasks} = $tasks_schema->list({ plan_id => $id }, { order_by => 'task_id' })->{items};
    return $plan;
}

Zapp also has a table for recording every time a plan is run, called "runs". This table also records the tasks and inputs the plan had when it was run in "run_tasks" and "run_inputs" respectively (along with some additional fields to record the status of the tasks and the user's actual input). In a way, a run is a plan. As above, when a user looks at a run, they need to also see the run's tasks and inputs. Since Yancy::Model uses plain Perl objects, I can refactor the plans class to also handle runs.

package Zapp::Schema::Plans;
use Mojo::Base 'Yancy::Model::Schema', -signatures;
has tasks_table  => 'plan_tasks';
has inputs_table => 'plan_inputs';
sub get( $self, $id, %opt ) {
    my $plan = $self->SUPER::get( $id, %opt );
    my $inputs_schema = $self->model->schema( $self->inputs_schema );
    $plan->{inputs} = $inputs_schema->list({ plan_id => $id }, { order_by => 'rank' })->{items};
    my $tasks_schema = $self->model->schema( $self->tasks_schema );
    $plan->{tasks} = $tasks_schema->list({ plan_id => $id }, { order_by => 'task_id' })->{items};
    return $plan;
}

package Zapp::Schema::Runs;
use Mojo::Base 'Zapp::Schema::Plans', -signatures;
has tasks_table  => 'run_tasks';
has inputs_table => 'run_inputs';

Now plans and runs both fetch their related data automatically. I can add similar functionality to create, set, list, and delete to make dealing with the related data seamless. Further, I can create completely custom methods like enqueue which will add the plan (or the run) to the Minion job queue to be executed.

With Yancy::Model I can quickly build an API for my application's data. As my application develops, I can keep my data logic separate from my web frontend logic. Since they're separate, I can use my data logic in other applications and tools. This is the biggest benefit of using an MVC pattern with a framework like Mojolicious.

Zapp: Runbook Automation

Doug Bell — Thu, 29 Jul 2021 00:08:12 +0000

preaction / Zapp

Create plans for your Minions to run

I hate runbooks. I've been an SRE from since we were called webmasters. In all this time, the least helpful part of my job has been the lists of commands to run when there's a problem. Often out-of-date, frequently lacking in context, and altogether useless when production is burning around you and clients are banging on the door. I'm a programmer. I don't perform tedious tasks, I automate them. Why, then, should I like documenting tedious tasks?

I hate Jenkins. No, that's not quite true. I'm disappointed in Jenkins. Parameterized builds could be such a useful tool, but the input form they provide the end-user is ugly at best, incomprehensible at worst. Jenkins's pipeline syntax opens the door to some truly amazing abilities, but there's no way for me to package those abilities in a way I can simply give to someone without training them on how to navigate Jenkins's UI.

To solve these problems, I wrote Zapp. In Zapp, I can create a plan as a series of tasks, like a Jenkins build. Plans can have input fields, like a Jenkins parameterized build. Users can then run the plan and see the output.

Zapp tasks can be more than a shell command or script. Zapp tasks have their own configuration, so I can provide a friendly form for the user to control what the task will do. Additionally, tasks output typed data that can be used as input to subsequent tasks. When the task executes, it can render its own output to show the end-user what is happening.

So, I can create a plan that asks a user to input their credentials.

Then, the plan can fetch an authentication token using those credentials.

Finally, with this authentication token, I can perform my request.

Then when I want to run the plan, I can input my credentials and hit a button.

As the job runs, it updates the job page. If any errors occur, the job stops and the failing task shows an error message. I can choose to re-play the job, or I can go back and fix whatever problem there is.

Zapp isn't limited to automating runbooks. Zapp provides webhook triggers to perform plans automatically. Build your software after a commit is pushed to Github, respond to Slack commands, or (when a future version adds scheduled tasks) run a nightly SQL report. Not only are these tasks automated, but they are easy to configure so that users can support themselves.

Finally, if Zapp can't do what you want, you can write your own task classes to perform tasks, type classes to accept input from users, and trigger classes to run plans automatically in response to events. Zapp is written in modern Perl using the Mojolicious web framework and the Minion task runner.

With Zapp, I can make tedious support processes into a single web form, or even a single click. As needs change, processes can be tweaked using a friendly UI. And new types of tasks can be created to provide ever more options for your users.

Zapp is very much Alpha-grade software right now: I use it personally, but there are bugs and missing features yet. The plugin APIs are pretty solid, though, so you can write your own tasks, types, and triggers. There are a lot of features that Minion provides that Zapp does not yet take advantage of, so there's plenty of possibilities for future development. If you have any questions, suggestions, issues, or want to know if Zapp can work for you, feel free to let me know at doug@preaction.me or on irc.libera.chat in #mojo-yancy.