DEV Community: Jem

Graphile Worker 0.16 released — reliable, extensible and performant

Jem — Mon, 11 Dec 2023 15:08:59 +0000

Worker 0.16 is here — freshly updated with optional TypeScript typing for tasks, a public jobs interface, a new plugin system, and much more besides!

The new plugin system enables ESM and CommonJS to peacefully coexist, adds support for writing tasks in import()–capable compile–to–JS languages like TypeScript, adds support for writing tasks in alternative languages (e.g. Python, Ruby, Rust), and could open the door to a lot more customizability in future! Version 0.16 also introduces support for “Worker Pro” — a plugin available to most Sponsors which monitors running workers, releases locked jobs workers that exit uncleanly, helps ease migrations to new releases, and will add more quality–of–life improvements in time.

We’ve backported a large amount of v0.16’s functionality to v0.13 and v0.15; this includes plugin and Worker Pro support to ease migration to the new version.

Thank you to each and every one of our sponsors, together you have made this massive release possible!

What’s new in Version 0.16?

Graphile Worker gets a logo!

Inspired by the Graphile heart and the PostGraphile elephant, Graphile Worker gets its new heart–shaped worker–ant logo. We admire the industriousness of rainforest leaf–cutter ants, relentlessly working away traversing miles of branches to bring food and messages back to their colony. What better animal to represent the spirit of Graphile Worker?

New Graphile Worker website!

As Graphile Worker's featureset has been growing, managing it all through a README has become challenging, both for users and for maintenance. We've moved all the documentation to a new website and spent some time overhauling it with new tips and notes (and corrections). Check it out at https://worker.graphile.org

Graphile Config and the new plug-in system

Worker is now optionally configurable with Graphile Config. This enables a whole suite of new options and features, not the least of which is you can now share preset files across multiple projects! Graphile Config takes everything we’ve learned from the PostGraphile project relating to configuration and plugins and expands these capabilities into a generic system that Worker can leverage. The plumbing has been completed, the initial hooks are there and ready to be used by everyone to make Worker even more adaptable and powerful than ever before. If you need to do something that there’s not a hook for yet, adding a suitable one is likely straightforward… get in touch for guidance!

Native ESM support!

The first thing we did with the new plugin system was to move the code for loading a task into its own plugin, and add support for loading both CommonJS files and ESM files. Since it’s a plugin, you can replace it entirely, loading tasks in any way you see fit!

Compile–to–JS language support!

With the plugin mentioned above you can add more file extensions to your config for any “compile–to–javascript” language that can be import()–ed — just be sure to make the relevant “loaders” available, e.g. for native TypeScript support you might use NODE_OPTIONS="--loader ts-node/esm" npx graphile-worker

Tasks in even more languages!

And the fun doesn't stop there! We've added another built–in plugin that allows us to offer support for writing task executors in other languages entirely: python, bash, even fortran; choose whichever language makes the most sense for the task at hand. All you need to do is place an executable file in the tasks/ folder and ensure it's named with the task identifier (extensions are ignored).

abortSignal

We have added the JavaScript pattern of abortSignal — a new ability to tell tasks to abort during shutdown. This means no more waiting for the task to complete during a graceful shutdown, so long as your task executor remembers to listen for the abortSignal your job can exit quickly when the worker is shutting down, and boot up again soon thereafter on a different worker.

Opt–in TypeScript typing of tasks

Now you can take down the guard rails, and assume the job type using the new GraphileWorker.Tasks global interface. This is not for the faint of heart, and in fact we don’t generally recommend it! But it has been requested many times, and now it is available to use via a public–facing declaration–mergeable interface. Declaring a global task interface allows you to use auto–complete and payload safety for addJob and quickAddJob, and specifies the assumed types of the payloads of your task functions. More, including important caveats, can be found in our documentation.

Remember the caveats of what it means to assume…

Public jobs view

This release makes it much clearer that the tables in which jobs and queues are stored are private implementation details — Graphile Worker needs to be able to change these tables to add new features or perform optimizations. For those users who look into the jobs table, even though we advise against it, we've created a public view that presents a stable interface into the queued jobs. We strongly advise against polling this, it will impact the performance of your worker, but it is now more suitable for usage in automated scripts since the interface is stable.

Worker Pro: upgrade your experience!

With thanks to each and every sponsor (past and present) for reinvesting back into open source, the growth of our community–based funding over the last six years has enabled us to spend more and more time maintaining and improving our various open source packages. These past few months we're grateful to welcome Trigger.dev and Seam as the latest in the organizations reinvesting in the software they use — thank you!

Alas, a significant amount of our maintenance and development time is still self–funded; so in an attempt to make this project more sustainable, we have introduced the proprietary Worker Pro plugin, and initially made it available to sponsors on the $25/mo tier and above.

Worker Pro adds worker tracking to Graphile Worker, making it a clean separation from the open source version which does not track the individual workers. With Worker Pro, each worker checks in with a "heartbeat" every so often, and we can use this in various ways.

For example, we have had feedback from the community that having to stop all workers in advance of updating to a new release is not ideal. With Worker Pro, before performing a breaking migration a graceful shutdown of all old workers is triggered, and further old workers are prevented from starting up. Once Worker Pro has confirmed all old workers have completed their graceful shutdowns, the migration can go ahead with significantly reduced risk of conflict. This avoids the need to “scale to zero” as all workers will communicate with each other to help the system as a whole remain consistent, and the newly running workers can gracefully take over from the old.

This new plugin also helps with crashed worker recovery: once a configurable amount of time has passed without receiving a heartbeat from a registered worker, Worker Pro can assume that the worker is no longer running (power pulled, killed, crashed, etc) and release all of its locked jobs to be re–attempted.

There are a couple more major features planned for “Worker Pro”, and once they are added it will be available via the $100 sponsorship tier. Now is a great time to sponsor Graphile and lock in this awesome 75% discount launch offer for at least 18 months!

To get access to “Worker Pro”, please reach out to Benjie or Jem on Discord or email team [at] graphile [dot] org — for the introductory offer you must ensure you have access before the price goes up!

Join our community — we run a Discord server which can be used for community support and to chat to others using the Graphile suite of developer tools. Find out how to sponsor us on graphile.org.

Full release notes found on the new Worker website or at GitHub.

Thank you Graphile sponsors!

Dev tools for databases, what is out there?

Jem — Wed, 19 Aug 2020 13:50:08 +0000

I've been spending a lot of my time this month thinking about databases and dev tooling. What's out there in this space? How do you know you're getting the best performance or security out of your database?

Get Started with Graphile

Jem — Fri, 13 Dec 2019 14:31:41 +0000

Last week, Benjie and I were very excited to unleash Graphile Starter on the world. This project is a gathering of many of our GraphQL & Postges tools, paired with Next.js and the AntD design framework to give developers a solid foundation for their React GraphQL apps.

In next to no time you can get a basic app running, with React routing, a user account system, a basic design framework, the beginnings of a Postgres database with associated migration system and much, much more - all under the MIT license so you can take what you want and continue building as much as you like without having to worry about any restrictions.

If you're already familiar with React, then this is a great project to take and play with. You can use it for a fun side-project or a hack day, but it's also ready for agencies to quickly build their SaaS apps or client projects.

The entire project is hosted on GitHub, take a look below:

graphile / starter

Opinionated SaaS quick-start with pre-built user account and organization system for full-stack application development in React, Node.js, GraphQL and PostgreSQL. Powered by PostGraphile, TypeScript, Apollo Client, Graphile Worker, Graphile Migrate, GraphQL Code Generator, Ant Design and Next.js

Graphile Starter

Take it for a spin!

We're running the starter at:

https://graphile-starter.herokuapp.com

Feel free to register an account and have a poke around as you see fit.

NOTE: emails are sent from Graphile Starter, so please only enter email addresses you control.

NOTE: every time we merge to master, we ship a new version of Graphile Starter to Heroku and entirely wipe the database, so your data may not persist If you wish to delete your data before this time you can do so via the delete account feature baked into the starter.

NOT FOR BEGINNERS

We do not advise that you build your own projects on top of this project until you're comfortable with the various tools it uses (Node.js Express, PostgreSQL, GraphQL, PostGraphile, Graphile Worker, Graphile Migrate, TypeScript, React, Apollo Client for React, GraphQL…

View on GitHub

Let's have a look at how to get it up and running! For this guide I'll be using VSCode, as the Starter has some handy editor and debugger settings as well as plugin recommendations built in.

Download and set up

First, visit the GitHub repo and download the latest zip file from the "Releases" tab. This tab is sometimes hard to see, I often spend minutes looking for it and I help maintain the Graphile repos! You'll find it nestled in all the tabs at the top of the file list.

Extract the zip file into a new folder and then open it in VSCode. You'll see VSCode give a couple of warning dialog boxes, but these can be ignored to begin with.

Initialise version control

Open a terminal pane and add version control to your project by running:

git init
git add .
git commit -m "Graphile Starter base"

Install yarn

Next, you need to install yarn if you don't already have it. Yarn is similar to npm - in fact installing yarn using npm is akin to downloading Firefox using Microsoft Edge. Graphile Starter uses yarn as it has smaller overheads and leads to better performance in the project. Install yarn using npm install -g yarn.

Run setup script

Now is the time to set up your Graphile environment by running yarn setup. This command will take you through a few questions to set everything up for you. It'll ask for a database name, a hostname, and a superuser connection string. All of these have defaults so you can just press enter for each question, if you like.

A few set up questions in your terminal, the answers you give are bright blue and the default answer is grey

After running the script, you'll see that a node modules folder has been added to your file list, and the /data/ folder will show that it has been modified - this is where the database schema has been updated with your answers to the setup questions.

Address VSCode dialog boxes

You'll notice that VSCode opens some dialog boxes:

This one is about using Docker with the Starter project. To continue using the Starter locally, you can press "Don't show again". The project includes a docker folder with some setup. If you're interested in this, visit the Docker read me on GitHub.

This one says there are VSCode plugin recommendations associated with the project. If you press "Show Recommendations", a sidebar will open with all the listed plugins. You can then choose which of these to install by pressing the green install button.

The recommendations are:

ESLint
Debugger for Chrome
Prettier
Apollo GraphQL
Remote Development (only for Docker development)

These are all handy for formatting, highlighting and debugging your code. They aren't strictly needed, but they will enhance your experience of using the Starter project within VSCode.

Start your server

Run yarn start in your terminal.

After a little while, you'll be able to visit your localhost address in your browser and see the Starter app in action!

You'll see that the Starter already includes a user account system, including things like sending out verification emails. There is a design framework included, known as AntD and the React routing and server side rendering is handled by NextJS

Visit /graphiql and you'll get the PostGraphiQL IDE - a handy tool which lets you explore your GraphQL schema. There is already a very basic one included which handles the user accounts system. As you write your application's schema, you'll be able to visually explore it and draft GraphQL queries and mutations using this tool.

Make it yours

Now you're all set to make this project all yours! Take some time to read through the extensive readme or some of the pointers on the localhost version of the project homepage. Next you'll want to:

Change the project name in package.json
Change the project settings in @app/config/src/index.ts
Replace the README.md file with your own
Add your own copyright notices to the LICENSE.md file
Commit to git as you usually would
✨Sponsor Graphile 😉

We are so excited to release this project out into the world. If you want to get in contact about anything, please leave a comment or join our supportive Discord chat server.

I hope you enjoy this project, and 🥂here's to rapid development without the boilerplate🥂