Introduction
Here at Lexis Solutions, we deal with complex projects every day that involve scheduling and micro-service coordination. Until now, we had tried different solutions for job processing, but somehow we always felt that something could be done better. That’s why we decided to develop our technology and make it open source so everyone could benefit. This article explores how and why we created a new open source job queue dashboard - Agenda Admin.
What is a job queue?
A job queue contains a list of jobs waiting to be processed by a subsystem. It's the first place that a submitted job goes before becoming active. The job is held in the queue until some factors are met.
There are many job queue implementations for Node.js, including Bull, Bee, and Agenda.
Let's take a closer look at Agenda.
Agenda is an open-source, lightweight job scheduling library for Node.js, with many features like job priority, concurrency, and repeating, which uses MongoDB for the persistence layer.
One of the main issues is that we have to write code whenever we want to create a new job or view the old one. While it's not hard, it is slightly inconvenient. That's why we need a dashboard with all the basic functionalities, like a list of all jobs or creating/deleting jobs.
Existing Solutions
So far, there has been only one solution to this problem - Agendash, an open-source dashboard for Agenda. It allows us to schedule new jobs, requeue or delete old ones and search old ones, among other things; however, a few improvements could be made.
While the UI/UX could also be improved, the main issue was that many pull requests took too long to merge. Because of that, some basic functionalities, like deleting or requeueing jobs, were buggy or not working.
Note: As of writing this article, the most crucial bugs are already fixed on Agendash. Kudos to their team 💯
Our Solution
Our goal with Agenda Admin was to create a dashboard with a clean and easy-to-use UI and all the essential functionalities we needed to make working with jobs easier:
- jobs status, refresh interval is 15 seconds by default
- search jobs by name, including autocomplete
- filter jobs by metadata and status
- view job details
- delete and requeue jobs
- schedule a new job.
The user can see a list of all jobs, with the basic information for each one - their status, name, the last time the job was run, and when it will be executed again. Filtering the list was also a critical feature, with options to filter the jobs by their name, status, or some of their other properties.
The Architecture
For this project, we went for the mono-repo architecture, with the two main subprojects being the api, written in Node.js, and the client application, written in React. This approach has many advantages, including a shared build process for both the api and the client and enforcing code style standards across the project.
We used yarn workspaces and the mono-repo architecture to improve our dependency management. This allowed us to avoid unnecessary bloat by managing the dependencies across the whole project instead of having separate node_modules folders for each package.
The Development Process
We used Linear to track our tasks and bugs to make the development process more efficient. Linear's integration with GitHub was also convenient, allowing us to link every pull request to the corresponding ticket.
Of course, it was also essential for the whole team to be up to date on the progress, so we would meet regularly to discuss what had been done and how we should proceed with the upcoming tasks.
While not the most fun part of the project, we had to ensure that every app function was covered with tests to avoid any bugs. And to make catching bugs even easier, we automated the testing process with GitHub Actions. By automatically running the tests on every pull request, we could be confident that every regression would be caught before it reached production.
However, unit tests can only take us so far. While we could catch bugs in the logic, we still had to ensure there were no issues with the user experience. This is where we had help from the rest of the people at Lexis Solutions, as they would constantly test the app and report any inconveniences, so we could fix them and provide the users with the best experience possible.
We were also able to automate the application's deployment, including tracing the version number and publishing the package. This was achieved by using semantic-release and, again, GitHub Actions. Also, by following strict commit message conventions and prefacing every commit message with a specific flag depending on the changes, semantic release could quickly figure out the following version number. By creating an automated workflow for publishing the package, every time a branch was merged into the main, a new package would be published on NPM.
Conclusion
Now Agenda Admin is available as an NPM package, and as an open-source project on GitHub, at lexis-solutions/agenda-admin. Everybody is welcome to suggest new features or contribute; don’t forget to star it.
Stefan Mitov - Full-Stack Developer at Lexis Solutions
Top comments (0)