DEV Community: Tom Cafferkey

I implemented a queue system in NodeJS

Tom Cafferkey — Mon, 20 Apr 2020 05:11:54 +0000

It's common for workloads that can be processed asynchronously to increase in your application flow. This is the situation I found myself in building EmployRemotely.com (for context, this is a platform we created to help developers find remote jobs). Upon a user being registered or an advert being submitted I was performing various actions that didn't need to be performed immediately, such as:

Sending a user email
Distributing that advert to various channels (Twitter, Reddit etc)
Sending data off internally (tracking systems, Slack channels etc)

All of this was being handled within the relevant APIs, and as this list of actions grew it became more obvious this wasn't the right approach to take. It was becoming more complex, error prone and potentially providing a bad experience for the users. So I decided to look for a better alternative.

To avoid any confusion with the terminology used in this article here are the differences between the terms "adverts" and "jobs" mentioned throughout.

Advert - A job advertisement you would typically see published on the website to attract and inform candidates to apply for a specific position at a company
Job - A task that gets pushed into a queue system to be processed at a later stage. This can be anything.

Queues

A queue system is a way of storing enough information about a particular job for it to be carried out at a later stage. Usually the main app will store this information, and push it into a queue of jobs to be processed in the future.

Some of the benefits of a queue system include:

Distribute the workload over time
Decoupling work
Retry logic
Consistency between external system integration

In my case, if I wanted to distribute an advert to Twitter and/or Reddit I would add these to a queue, which would allow me to process them independently, in isolation which is decoupled from the original application flow.

Bull

After some research I decided to reach for Bull for my queuing system. Bull is "the fastest, most reliable, Redis-based queue for Node".

Bull simply ticked some important boxes for me. It was feature rich, had a very simple interface and looked easy enough to get up and running with. Because EmployRemotely.com is not full-time for me, time was definitely an important factor.

Implementation

1. Installation

a. Ensure you have Redis installed on your local machine.
b. Install Bull into your project npm install bull --save

2. Structure.

It always helps me to understand how things are tied together when I can see a directory structure. So, I created two new directories to separate queues and jobs.

3. Create queues and processes.

Witin the /queues/distributeAdvert.js file create my queue and processing function for each job to be processed by. The Queue constructor creates a new queue that is persisted in Redis. Every time the same queue is instantiated. The first parameter of the queue is the queue name.



// src/queues/distributeAdvert.js

const Queue = require('bull');
const sendTweet = require('../jobs/twitter');
const sendReddit = require('../jobs/reddit');

const distributeAdvert = new Queue('distributeAdvert', process.env.REDIS_URL);

distributeAdvert.process(async job => {
    const { slug, service } = job.data;

    try {
        switch (service) {
            case 'twitter': {
                const response = await sendTweet(job);
                return Promise.resolve({ sent: true, slug });
            }
            case 'reddit': {
                const response = await sendReddit(job);
                return Promise.resolve({ sent: true, slug });
            }
            default: {
                return Promise.resolve({ sent: true, slug });
            }
        }
    } catch (err) {
        return Promise.reject(err);
    }
});

module.exports = distributeAdvert;

4. Adding jobs to the queue.

In my API where I would handle the advert submission and create a database entry for it. Its here I was also previously sending this off to Twitter and Reddit to be published also.

Now I can remove these requests to Twitter and Reddit and simply replace it with the queue system we've created by adding the necessary job information to the queue to be processed.

Here the job is added to the distributeAdvert queue. The job is nothing but an Object that contains the required data to process it.



const express = require('express');
const { distributeAdvert } = require('../../queues/');

const router = express.Router();

router.post('/create', checkUser, async (req, res, next) => {
    const {
        ...
        slug,
    } = req.body;

    // ...code to insert advert into database

    try {
        distributeAdvert.add({ service: 'reddit', slug });
        distributeAdvert.add({ service: 'twitter', slug });

        return res.status(200).json({
            message: 'Your advert has been submitted successfully, good luck in your candidate search!',
        });
    } catch (err) {
        return res.status(422).json({
            message: 'There was an unexpected error submitting your advert.',
        });
    }
});

module.exports = router;

And that's all that is needed.

We've created our directory structure within the project
We've created our distributeAdvert queue
We've replaced requests to third parties (Twitter, Reddit etc) with code to add these jobs into our queue to be processed.

Summary

So in summary, by implementing queues, I have now:

Smoothed out my process
Decoupled unnecessary tasks from important APIs
Have a less complex and more readable process in place
Have an approach that will scale better
Made distributing an advert to third parties more consistent

Thanks for reading.

If you're interested in following our progress on EmployRemotely.com, including what works and what doesn't, head over to my Twitter @codebytom

10 myths I busted working remotely as a developer.

Tom Cafferkey — Tue, 14 Apr 2020 06:20:46 +0000

Before starting I think it's worth mentioning I am not a self-proclaimed remote working "expert". I'm just a developer who works remotely documenting my personal experiences of the transition from office to remote work.

Working remotely comes with its pros and cons, like everything. With so many different companies adopting remote work in so many different ways means that one companies problem does not necessarily mean it will be the next companies problem. That does not mean these obstacles don't exist, they do. But that is not what this article is about, this is about busting those myths you hear, often from people (possibly your managers) who say it just will not work for them or it "is not possible" for the following reasons.

Myth #1: Working remotely means working in isolation

One of the huge benefits of working remotely means you get to decide on any particular day where you feel the most productive working from based on your schedule. This might mean "deep focus" and a distraction-less environment isn't necessary. So you can take yourself off to a co-working space or coffee shop for the day to catch up with other likeminded freelancers, creatives or remote workers.

Although, you don't necessarily have to see people in person to avoid the feeling of loneliness or isolation. Scheduling in catch-ups with colleagues for those "water cooler chats" you would otherwise be having at the brew station in an office is just as important to maintaining and creating lasting relationships.

Myth #2: You can’t work remotely as a Junior

I often hear people being afraid of working remotely as a junior developer because they're afraid they just aren't good enough to work independently on a project. Personally I think that if this is an issue then it's not the employees issue but rather a failing of the company who hired you (it's also worth mentioning that this issue exists in an office too).

If you're looking for a job as a junior developer, or a company looking to hire a junior it's important to set these expectations early on. A new job fills you with anxiety at the best of times, everything is new and you're finding your feet while trying to play catch up with those around you. Unable to get that immediate feedback from tone of voice, body language or facial expressions in person as an indicator on your progress leads to you asking questions like "Am I good enough?" or "Did they expect me to pick all of this up quicker?".

There are solutions to this problem. A thorough and thoughtful on boarding process, documented workflows and processes as well as pair programming sessions with frequent check ins to remind them that they're on the right path. With positive encouragement from your team leader and/or mentor will replace feelings of failure and anxiety with self confidence.

Over at EmployRemotely.com we are thinking of ways of encouraging remote companies that hiring a junior is worth it. If you have any ideas around this then please get in touch.

Myth #3: You lose company culture online

When I moved from an office to a fully remote position I was concerned about this point myself. My office culture was brilliant and I did wonder if it was possible for it to be emulated online.

I wouldn't say that culture online is no better, or no worse than it is in an office. Culture starts from the top and as long as you have great leadership and management encouraging a positive culture everyone will follow.

Some things to encourage positive culture and engagement amongst your online team are:

Have public channels to discuss non-work related topics. This will help you figure out the interests of your colleagues (Yeah but who's watching Tiger King?)
Encourage humour in the form of gifs (who doesn't love a gif?), articles, videos etc.
Have optional company challenges. Things like step challenges, virtual run/walk/hiking clubs etc (Strava is great for this)
Knowledge sharing & show & tell sessions whether it be project work or anything else of interest.

Following these few suggestions might make it easier for you to engage with everybody and not just the people in your team.

Myth #4: You can’t build relationships if you don’t meet them

How can you really be friends, or build a relationship with someone having never met them in person? Easily.

Whilst I was working in an office I found it harder to get to know the people who I wasn't sat immediately next to. Whilst working remotely one thing that stood out was I never had that issue because was equidistance away from everybody in the company. Conversations within public Slack channels allows me to get an insight into their interests and humour which helps spark conversations you would never have known about working in an office. We nurture this by getting paired randomly with colleagues in the company weekly for a 30 minute chat. This has had a great response.

You may also find that you can have more thoughtful conversations online when they're needed. Being able to take a moment to type out a considered response goes a long way and your colleagues will appreciate that.

Myth #5: It’s much harder to work as a team

I don't think team work has been effected whilst working remotely. If there are issues, they would have existed in an office. But I am going to touch on some points which I think are important for all companies but one thing I cannot stress enough is that documentation is key.

Here is a list of documentation that I often look for

Company strategies and visions
Workflow processes
Project documentation (Getting started guides, acceptance criteria, known issues)
Feature documentation
Publicly asked questions and answers on outstanding work
Project analysis
Meeting notes
Everything!

Documentation is an important part of enabling your employees to work independently, removing dependencies on other people for their knowledge is something all companies will benefit from. Should individuals have any issues then daily stand ups are a great way to raise these and provide an opportunity to reach out for help or reassess goals.

Couple this with meetings to establish weekly or bi-weekly objectives, goals and responsibilities draws a clear line in the sand of what is expected from you.

Myth #6: All remote workers work in a hammock

Although this might have some element of truth to it when it comes your typical "nomadic remote worker". Largely, the majority of remote workers often have a professional workspace setup in their home.

One of the huge benefits of remote working is being able to decide, based on your schedule where you feel most productive working from whether that be in your home office so you can focus, or a co-working space. It's entirely up to you to change your environment to benefit your work. For myself, I'd say 80% of my time is spent in my home office.

Myth #7: You can “get away” with doing less

Another misconception is that employees that aren't working under the nose of their managers can get away with slacking off. It becomes obvious when someone working anywhere isn't pulling their weight. I'm guessing you don't measure employee output by keeping your eye on your team members screens for 8 hours a day in an office? No? Same goes for remote workers.

A benefit of being able to have a little bit more control over your own schedule is being able to take smaller breaks when you feel like your productivity is taking a hit. Although frowned upon in some offices, the time you spend focusing on work will likely result in a higher output. There are many techniques out there for this, the Pomodoro technique being one of the more popular ones.

Myth #8: Personal development takes a hit

Have you ever heard the phrase "If you're the smartest person in the room, you're in the wrong room"? Well, often working from home means you're the only person in the room and this can be a worry if you're concerned about not being surrounded by more talented people to learn from.

In my personal experience I have learnt more being on my own, by not having that immediate crutch in the form of a colleague to lean on for advice or help means I've had to put more thought and effort into solutions myself. Also with the addition of code reviews, pair programming sessions (VS Live Share is great for this) and course material it means my personal development has increased rather than decreased.

Mindset also plays a part in self development, and with all that time I have saved in cutting out my commute it means I can spend it more productively. Often spending it studying, doing exercise or walking my golden retriever Indie means I start each day in a clear mindset. After-all, you can't teach someone who doesn't want to be taught.

Myth #9: It’s harder to lead a team online

I often have these conversations with people who can't wrap their head around distributed teams. Questions I get asked are "How do you know they are working?" or "How can you lead by example if they cant see you working?"

With good planning, retrospectives and project reviews you are easily able to identify areas where improvements can be made. And also areas to continue nurturing. Checking in with team members provides opportunities to discuss how they're coping as well as their aspirations as a developer so you can offer support and guidance where needed. It's important to regularly review teams progress and happiness and I believe a lot of the points we've touched on will help this. But ultimately, working remotely means giving people more ownership and autonomy, micro management will not work here.

Myth #10: It didn’t work, we tried it during the pandemic

It would be a shame to hear companies say "It didn't work, we tried it during the pandemic" as an excuse for not adopting a remote working culture. A lot of companies were not prepared for this situation we now find ourselves in. They have been scrambling to get systems in place to accommodate their employees working from home, meaning their IT teams have probably been working under intense pressure and round the clock to get these processes in place.

Speaking to employees within companies who haven't been prepared have said they felt a sense of it being rushed which can have a mental impact on the team. If the leadership is anxious and worrying that will naturally filter down to the employee hierarchy.

Additionally leadership are now having to take extra steps to check in on their team to make sure they have enough support, are feeling OK working in isolation when they would otherwise be spending that team on other tasks. This I can imagine is extremely hard when you've got to manage a remote team with zero remote experience yourself.

Finally lets not also forget that we are in a constant state of isolation due to the national lockdowns worldwide, this can have a serious mental impact as they are not able to get out the house to integrate socially within their communities which is a huge benefit of working remotely.

Looking for your next remote developers position?

Over at EmployRemotely.com we are focusing on helping developers specifically find their next 100% remote opportunity. For candidates, we also run RateThisCV.com to help job seekers get free feedback on their CVs/resumes before applying for positions.

During this time of economic uncertainty, we have made all our services completely free. We have seen too many passionate developers lose out on opportunities recently and want to do everything we can to help those during this difficult time.

If you're interested in how and why we built EmployRemotely.com then head over to my previous article "Why and how I built a platform to help developers find remote jobs"

Why and how I built a platform to help developers find remote jobs.

Tom Cafferkey — Mon, 06 Apr 2020 07:43:13 +0000

Working remotely is becoming more and more popular. Since 2005 it has grown 173% and continues to increase year on year. This growth can be attributed to many factors including technological advances in communication, the rising cost of property rental in cities, and the ever-growing list of benefits that working remotely offers to both employees and companies.

With this growth comes opportunities. Companies hiring remotely now have the world's talent pool to pick from. For employees, there’s the opportunity to work for amazing companies anywhere in the world. All that is required is the relevant skills and an internet connection.

So, why another remote jobs board?

Job boards often cater to a vast array of professions, which makes it difficult for a software developer like myself to filter out the job openings that are relevant to me. I found that it took unnecessary effort to discover jobs which used technologies I knew, was interested in learning, or had past experience in using.

Equally, watering down hundreds of job adverts into a small list of opportunities that were relevant to me was time-consuming. And after all that filtering, finding out these positions required 1 day a week commuting to an office in San Francisco, just wasn’t something I was prepared to commit to. Being based in Britain, that’s a 12-hour flight and a 2-hour drive once a week!

My free time was constantly being split between trawling these types of sites and their newsletters, which also contained opportunities just as irrelevant. For myself, a Front End Developer (depending on who you ask), I wasn’t interested in applying for a Junior DevOps Engineer position that required 8 years experience and 3 degrees (I don’t even have 1).

So I decided to try and improve things for myself and people in my position.

Finding a solution

I decided to team up with a friend of mine Nick, who also worked remotely and specialised in user experience. Having faced similar problems himself, he didn’t take much convincing that we could make some improvements in this area. So we set about putting a plan together.

Focus on a niche

Channelling our efforts into a niche meant we solved two problems. The first being that we could get a product of value out quicker. The second being that we weren’t falling into the same trap we saw other boards doing, which was catering for an audience that was too broad.

Organisation of jobs

Instead of listing all of these jobs under one umbrella term of “Programming” or “Software Development” we were able to categorize them based on the disciplines we had chosen so far which were front end, back end, full stack and mobile.

We also encourage companies to tag their job advert with relevant skills and technologies related to the position when posting a vacancy. This will allow us in future iterations of the product to return more granular results for our users.

Encourage transparency

Although we don’t force companies to publish all the information we would like them to, we are always looking at ways to encourage and educate companies on the importance of full transparency to job seekers. Putting in important information such as salaries, and interview processes will vastly reduce the number of irrelevant candidates you would otherwise waste time filtering through manually.

Improve the newsletter relevance

Just like we looked at solving the issue with categorising jobs on the site, we also needed to ensure that we were applying the same experience to newsletters. We didn’t want to be those guys who delivered irrelevant content to users who trusted us with their personal information. For this, we simply gave users the option to receive only the types of jobs they were interested in.

How we built EmployRemotely.com

I now realise that building a product around a full-time job is easier said than done. People have their own lives and responsibilities to manage, which isn’t as simple as it sounds. The same goes for us.

Collaboration and Design

For sharing designs, content, ideas and planning our work we primarily use 2 tools. Notion and Figma make it easy to organise and share designs and tasks, putting them into weekly sprints based on what we want to achieve depending on how our week ahead is looking. Sometimes we may only get a handful of hours to commit to working on our project but organising these tasks effectively means that we have focus and don’t depend on each other being available at all times for assistance.

Back End & APIs

For the backend of the website, we used NodeJS & ExpressJS. JavaScript and Node are languages I’m comfortable with and are suitable for a project like this. So in the interest of time, we stuck with these choices.

Front End

I’m a big fan of ReactJS and its flexibility in organising and architecting codebases as they scale. Because we also wanted the added benefit of server-side rendering, I reached for my go-to framework which is NextJS. The people behind NextJS do an amazing job of maintaining and documenting this framework and I highly recommend it to anyone interested in working with React.

For the CSS, I took an ITCSS approach to structuring my SCSS coupled with BEM methodology. Yep, I fall into the category of developers that do not write CSS-in-JS.

Database & the learning curve

For this project, I had to take a few courses on SQL and database architecture. Not just to up-skill myself but also to brush up on the little experience I have working with SQL. For this, I found Caleb Curry YouTube videos really helpful. My database is also hosted on AWS.

For interacting with the database I picked Knex.JS based on a friends recommendation. Although not used many alternatives of this for a compartive insight, Knex.JS has been easy to work with.

Testing

I don't think I'm the only one who isn't proud of their side projects test coverage. I haven't written as many tests as I'd like to but I do what I can with the time I'm given. For this I have chosen Jest and Kent C Dodds React Testing Library

Finding your next remote job

If you’re looking for your next remote opportunity in software development, then I urge you to check out our new website EmployRemotely.com and provide feedback to us. It currently contains 100s of opportunities from amazing companies and continues to grow by the day. For candidates, we also run RateThisCV.com to help job seekers get free feedback on their CVs/resumes before applying for positions.

For companies hiring remotely, we have made the site completely free to post your jobs on. Due to the economic uncertainty during this global pandemic, we want to do whatever we can to help at this moment in time.

As this product is relatively new, consider this the first in a series of posts as we continue to iterate and document our progress and learnings where they are of value to our readers.

Thank you for reading, and good luck in your job search.

If you're interested in following our progress, including what works and what doesn't, head over to my Twitter @codebytom