At my day job I lead the development of a product that makes heavy use of queue technology. We developed our own in-house solution that handles several million jobs per week with varying payload sizes, and adapting it over the years to these growing loads has been interesting and rewarding.
As a personal challenge, I've wanted to develop a full project with a language other than PHP, the one I use in my job. So I thought I'd develop a queue server using TypeScript, which transpiles strictly typed .ts files into JavaScript, giving you the security and durability of languages that are natively strictly typed.
Deno is a runtime that conveniently packages a lot of useful utilities for JavaScript developments, such as inbuilt TypeScript support, easy testing functions, and the ability to compile your project into a self-contained executable across multiple platforms. So it seemed like a great fit for this fun personal challenge.
Queues
Let's start with what sounds like a daft question: What is a queue? A queue is a data structure containing items with an order. You enqueue an item by appending it to the end of a queue, and dequeue an item by removing it from the start of the queue.
Here's a handy diagram from a post by Kirill Vasiltsov:
A FIFO ("First-in, First-out") queue guarantees this order, usually along with a guarantee that each item is only read and removed once - a particular need for applications such as customer email software, where spamming customers with repeatedly executing email sending jobs from a queue will not make us very popular as developers!
This is the data structure that the queue server project will need to model. It needs to be able to enqueue payloads and dequeue payloads, delivering the earliest first. Sounds simple, right? Well, it's always easy to say that at the outset of a project! 🙂 We also need to persist our queue data if it is to be taken seriously.
Deno
It's a slight spoiler for the end of this post, but I've really fallen in love with the Deno runtime while working on this personal project. It made every step of the development process enjoyable, and I will surely use it again in future projects. Writing code, compiling, and testing were all super simple thanks to Deno's expressive functionality.
As an experienced developer who has been burned by too many errors in production, I've become more appreciative of strict typing. Getting compiler errors while developing with TypeScript is a pain, but it is much better than experiencing the shame and embarrassment of production errors.
Deno can run .ts files without a fuss, running the type checker for you without any need for additional packages. If you run deno test within a directory, it will search for files that look like tests, such as test.ts or ./test/foo.ts and automatically run them, giving you a neat and friendly output with the results.
Writing a test in one of these files is as simple as calling an inbuilt function, automatically injected by the Deno runtime. I'm all for making tests both realistic and easy-to-write! Here's an example test from my queue server project, where I'm testing a queue manager class for the basic FIFO functionality described earlier:
Deno.test("manager enqueue", () => {
const mgr = new QueueManager(new Persistency.None);
mgr.enqueue("queue", "foo");
mgr.enqueue("queue", "bar");
assertEquals("foo", mgr.dequeue("queue"));
assertEquals("bar", mgr.dequeue("queue"));
});
Logic
The basic logic of the queue server is best approximated by the simple queue class I wrote for my project. Notice that it has the FIFO functionality I mentioned earlier: enqueuing payloads and dequeueing the earliest payload on the list of items. It's pretty simple, but I like simple!
export default class Queue<T> {
private messages: Array<string>;
constructor(messages: Array<string>) {
this.messages = messages;
}
public length(): number {
return this.messages.length;
}
public enqueue(payload: string): void {
this.messages.push(payload);
}
public dequeue(): string | undefined {
return this.messages.shift();
}
}
I wrapped up this functionality in a QueueManager class which takes Queue as a parameter, acting as a useful interface between clients of the queue server and the individual queues that the server will manage. It's a subtle addition, but it makes a big difference.
Here's a demo of it responding to cURL requests. Notice that the /dequeue/:queue endpoint returns the earliest payload first, as we wanted!
Persistence
For people to take this queue server seriously, it needs to persist queue data, even when the queue server is turned off and on again. For business-critical applications, we can't afford to lose data, so I couldn't quit before adding persistency to this project.
Initially, I thought about manually appending new items to a file and reading items from a file in a synchronous manner, every time an action was requested by a client. But that was a nightmare, so I tried a different approach: binary logging.
Binary logging means keeping a log of every write event in a file as it happens. Then, when the server is turned off and reloaded, it can replay all the events from the binary log file, causing it to be in the same state as it was before it was switched off.
Fortunately, Deno makes this just as easy as the other steps of the development process. I defined a TypeScript interface that my QueueManager class could interact with, and implemented it with a class that contained the functionality to read and write to a binary log file:
export class File implements Persist {
private directory: string = '';
public append(line: string): void {
Deno.writeFileSync(this.directory + "persist.dat", new TextEncoder().encode(line + "\n"), {append: true});
}
public clear(): void {
Deno.truncateSync(this.directory + "persist.dat");
}
public load(): string {
return new TextDecoder().decode(Deno.readFileSync(this.directory + "persist.dat"));
}
public dir(dir: string): void {
this.directory = dir.replace(/\/$/, '') + "/";
}
}
Notice that Deno comes with inbuilt utilities for handling files, and you can see other uses of these on the Deno by Example page.
The more astute among you might notice that I'm handling reads and writes in a synchronous manner, which can be inefficient when our files become large. It would be better here to make proper use of Deno's async/await functionality to make this part of the queue more efficient.
With a nice little test, we can verify that this core piece of functionality works as we expect:
Deno.test("manager persistency", () => {
const persist = new Persistency.File;
const mgr = new QueueManager(persist);
persist.clear();
persist.append(`{ "queue": "foo", "payload": "bar", "enqueue": true, "dequeue": false }`);
persist.append(`{ "queue": "fee", "payload": "bat", "enqueue": true, "dequeue": false }`);
persist.append(`{ "queue": "fee", "payload": "gat", "enqueue": true, "dequeue": false }`);
persist.append(`{ "queue": "fee", "payload": "bat", "enqueue": false, "dequeue": true }`);
mgr.load();
assertEquals("", persist.load());
assertEquals(1, mgr.length("foo"));
assertEquals("bar", mgr.dequeue("foo"));
assertEquals(1, mgr.length("fee"));
assertEquals("gat", mgr.dequeue("fee"));
});
When I finally got this test to pass, I had a massive smile on my face. This meant that my queue server not only worked as a basic queue, but it can also be used as a part of a serious platform requiring data persistency and stability.
Conclusion
I really enjoyed this fun personal project, and it made me fall in love with TypeScript and Deno. What seems like a simple runtime actually comes with a bunch of really useful and delightful utilities that make the development experience much easier and smoother.
I've published the project here: https://github.com/jonbaldie/queue. I encourage you to take a look.
If you'd like to try out the queue server, I also made a Docker image which can be used in this way:
docker run -d -e HOST=127.0.0.1 -e PORT=1991 jonbaldie/queue
Note the use of environment variables to change the listening address. I've written some documentation on the README.md file for the project, so you can see how to use cURL requests to interact with it!
If you enjoyed reading this, or found it useful, please let me know! I hope it inspires you to try out TypeScript or Deno, or to learn more about queue technology.
Top comments (0)