DEV Community: Geshan Manandhar

Set Up Tracing for a Node.js Application on AppSignal

Geshan Manandhar — Wed, 07 Jun 2023 14:41:58 +0000

In this post, you will learn how to add tracing to a Node.js application on AppSignal. You will use an existing Quotes app that talks to a PostgreSQL database to fetch the quotes.

Let’s get going!

Why You Need Tracing and an APM Tool: A Case Study

Let’s start with a story. You are pinged by your manager who tells you that the app you deployed yesterday has been reported as slow by users. Your manager claims that the complaints started yesterday at 2 pm, and you released your change at 1:55 pm. Without an Application Performance Monitoring (APM) tool or data trace, you are just grappling to understand what really happened.

Your tests on staging were fine, and you experienced no issues when developing with the small data set you had on your local machine. After almost two hours of digging around, you find out that due to the high volume of data in the production database, a couple of the queries you introduced yesterday caused the slowness. You push a fix, adding a couple of indexes to the problem tables, and the app is back to its normal response time again.

This is where the power of APM and tracing becomes critical, if not crucial to your applications (especially in a production environment). If you had an APM like AppSignal, you would know about the problem minutes after you deployed by checking your app's performance graphs.

If you had a trigger and alert set up for the throughput or response time, the issue would be caught minutes after deploying the new change.

Proper tracing can pinpoint the line of code and query that introduces a problem. In the next section, we'll set up tracing and APM for an example application.

Tracing and APM for a Node.js Application

You will set up tracing for an example Quotes app built with Node.js using Express. It fetches data from PostgreSQL hosted on ElephantSQL. The application, built with Node.js and Postgres, is open source and the code is available on GitHub.

The app is hosted on Render.com for free, and the database is also on a free plan on ElephantSQL. You can see an example of the quotes returned by the API below:

To follow this tutorial, you can clone the GitHub repository and copy the code to your machine with:

git clone git@github.com:geshan/nodejs-posgresql.git

Then run npm install to install all the dependencies. It has been tested with Node.js v18, the latest LTS at the time of writing. After that, if you run npm start and hit http://localhost:3000/quotes on your browser of choice, you will see a similar output to the above.

Now, let's add the AppSignal library to this Quotes app.

Add AppSignal to the Node.js Project

To add AppSignal to any project, you first need to register for an account. AppSignal offers a 30-day free trial with no credit card information required, so you can sign up without any issues.

After you sign up, you can follow the official Node installation docs to set up AppSignal in your Node.js application.

Run the Node.js App with AppSignal Reporting

As the library has been added to the app, when you run the app, it will start reporting data to AppSignal. The npx command creates an appsignal.cjs file on the project's root that connects your app to AppSignal. To include this pre-run script, you need to change the app start script to node --require './appsignal.cjs' ./bin/www in the package.json file’s start script.

Now if you run your Quotes app again with npm start and hit it with some requests, it will show up in the “development” environment of the project, which is taken from the NODE_ENV variable. It will look something similar to the below after a couple of requests:

Hooray! You have successfully added the AppSignal library to your app, which communicates data and metrics to AppSignal from your local machine.

In the next section, we will send some requests to the app.

Sending Requests to the Node.js App

One of the easiest ways to send lots of fabricated requests at the same time is to use the Vegeta load testing tool. Being a load testing tool, it can send lots of requests consistently, every second, to the given target URL. You can read more about Vegeta on GitHub. The binary can be downloaded and used without installation.

After you have the executable binary for Vegeta, run the following command to send 3 requests per second for 1 minute (60 seconds) to your local web server.

echo "GET http://localhost:3000/quotes/" | vegeta attack -duration=60s -rate=3 | vegeta report --type=text

It will show an output like the below after running for 60 seconds:

The report says that a total of 180 (3*60) requests were sent, and all came back with a status code of 200 OK. The fastest request took 184.803 ms, and the slowest one was 2.95 seconds. There are other interesting stats here too.

Multiple factors can slow down your application's responsiveness. Luckily, most of them are under your control and you can work on them to make your app faster. Some of the factors that you can influence to speed up your application include:

Balancing the resources allocated to the application
Code level optimization
Adding database indexes
Improving the SQL queries

Depending on the architecture and infrastructure of the project, you can also look into horizontal scaling and caching.

How you send the queries to the database can be an issue too. AppSignal gives you deeper insights on queries and the time they take to execute. We'll cover how to speed up SQL queries later in this post.

Analyzing Stats and Graphs

Now if you check your app's development environment in AppSignal after a minute, you can see the load handled by the app in metrics and graphs. You can check for memory usage, response times, throughput, and errors.

Checking Memory Usage

On the Node.js Heap Statistics dashboard, you can see that the memory consumption of this app while serving 3 requests per second was around 40 MB.

Checking memory usage is very useful to know how much memory your application consumes. Suppose your application gets a lot of traffic at the same time, and only 128 MB of memory is allocated to the application. The container/pod might be killed due to an Out Of Memory (OOM) issue.

Similarly, on the flip side, maybe your application is allocated 512 MB of memory, and, for weeks, it has only consumed 100 MB at its peak. Then it is time to allocate a lower amount of memory. This also saves costs, as 80% of the allocated memory was not used and will not be used for the expected traffic. The same logic can be applied to other resources allocated to the application, like the CPU.

Visualizing Response Times

You can also check the app's response times, throughput, and other stats while it was serving 3 requests per second (RPS) in 1 minute at Performance > Graphs. It looks like the following for the above Vegeta load test.

So, at around 180 requests per minute (the load sent using Vegeta), it responded at ~760 ms on the 95th percentile.

This means the slowest 5% of the requests responded to the client in 760 ms. Response times are very important for APIs and web applications. More than a decade ago, Amazon found every 100ms of latency could cost them $1.6 billion in sales. So slow websites and APIs are not great for your business.

Keep a tab on response times and do what's needed to optimize them. Response times can be decreased by either consuming fewer resources, like CPU and memory, or adding more resources to your servers/containers.

Response times can also be brought down by speeding up slow queries, and we'll discuss this next.

Know Your Slow Queries

You can pinpoint the slow queries in your application under Performance > Slow queries. For example, see the slow SQL query below.

If you know about slow queries, then you can fix them in multiple ways. You can use an EXPLAIN prefix to find out why SELECT queries are slow. It returns the execution plan generated by the query planner.

To oversimplify EXPLAIN, it does a full table scan to tell you if a query includes more rows to get results than needed (i.e., the query could have got the same rows by scanning fewer data). You can add indexes to relevant column(s) to make the queries faster. Indexes make reads fast, but they also make writes slow. So make an informed decision as to whether you should add indexes.

Another way to speed up slow SQL is by rewriting and refactoring a query. For instance, you may remove a join that isn't needed or convert a subquery to a join on a case-by-case basis.

Let's now turn our attention to tracking errors in AppSignal.

Tracking Errors

You can go to Error > Graphs in AppSignal to see your application's error rate. Depending on the errors or even response times, you can add triggers at Anomaly Detection > Triggers. Then set alerts on those errors.

For example, if your throughput is less than 100 per minute, you can trigger an email alert. If the error rate is more than 2%, you can trigger another alert, and so on.

Below is a code snippet from the quotes file in the example app, where an error has been deliberately introduced.

/* GET quotes listing error. */
router.get("/error-fix", async function (req, res, next) {
  try {
    res.json(await quotes.getError(req.query.page));
  } catch (err) {
    console.error(`Error while getting quotes `, err.message);
    appSignal.setError(err);
    res.status(err.statusCode || 500).json({ message: err.message });
  }
});

The quotes service does not have the getError method. This will always go into the catch section. As the appSignal.setError method is called with the error, it is sent to AppSignal. After the URL is hit, after some time the error will show up on the AppSignal interface under Errors > Issue list, as follows:

Then, you can click on the particular error in the GET /quotes/error-fix path. You will see the error summary and can click on the 'Samples' tab to see something like the below:

If you select one of the samples, you can see more details about that error sample:

Then if you scroll down to the Backtrace section and click the green button saying Show full backtrace, you can view exactly where the error came from (and what files and functions led to that function call), as follows:

Given this was a fabricated error, it will be fixed as soon as you call an existing method on the Quotes service. You can also view error graphs for different time ranges on AppSignal like the below:

Now, imagine debugging a pesky bug or a performance issue and having all this data, metrics, and stats at your disposal. It's like having a powerful torch in a dark room — you can find issues much more quickly and are one step closer to solving them.

Other Useful Features in AppSignal

Play around with the links on the left sidebar in AppSignal. You can add an uptime monitor. If your website or web application is down for some reason, you will be notified. Your app is pinged every minute from 4 different locations across the globe.

There's the Host metrics section too, which includes CPU Usage, Load Average, Memory Usage, and other host stats.

Wrapping Up

In this post, you first learned why tracing and Application Performance Monitoring (APM) is important. After that, we introduced an example app that serves Quotes via a JSON API. Then we added AppSignal APM and tracing to that example app.

We explored some of AppSignal's useful features for your Node.js app, including performance graphs, slow queries, errors, and response times.

Happy tracing!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

How to Handle Async Code in JavaScript

Geshan Manandhar — Wed, 23 Nov 2022 11:59:42 +0000

The easiest way to understand asynchronous code is to realize that the code does not execute sequentially. This can be difficult to comprehend in JavaScript, especially if you come from a programming language that's synchronous or sequential by default, like PHP.

In this post, you will learn how to write async (also known as 'non-sequential') code in JavaScript efficiently. You'll learn the basics of using callbacks, promises, and the modern async/await style.

Let's get started!

Async Programming: A Quick Intro

Let's start with the basics. There are two execution models in programming languages: synchronous and asynchronous.

The synchronous model is where the next line of code is not executed until the current one is done. Even if the current line of code calls an API that responds in 500ms or reads a 100 MB file, the execution will wait until the line of code fully completes. In other words, in synchronous execution, things happen one at a time, one after the other.

On the other hand, if there are five lines of code, and line two calls an API, it is pushed to the background with a mechanism that lets the main execution know that the API has responded.

While that is happening, lines three to five are also being executed. This is an oversimplified explanation of the asynchronous or async code execution model. In this case, the code does not run line by line in sequence. Things can be put in the background (or queue), and later the result will be known. This model of execution allows multiple things to happen at the same time.

The notion of the execution model is essential here because, depending on the programming language, it can work in a sync way or an async way. For example, in Python, you can write programs in a sync or async fashion — whereas JavaScript is asynchronous by default.

How? You may ask. First, let’s look at a basic example, with a simple synchronous execution showing logs:

console.log("first log line");
console.log("second log line");
console.log("third log line");

The above code is pretty straightforward and will print three log lines one after the other, like below:

first log line
second log line
third log line

Now, let’s make it a bit more interesting. See that JavaScript code does not execute sequentially with a 'waiting 1 second' example:

console.log("first log line");

// 1 second wait
setTimeout(function () {
  console.log("third log line - after 1 second");
}, 1000);

console.log("second log line");

Now, when the above code is executed, the output is as follows:

first log line
second log line
third log line - after 1 second

So what happens here? first log line is printed, then the setTimeout function is called, which executes a function or specified code only after the timer expires. It is set to 1 second — 1000 ms. So this code is pushed to the background to be executed after 1 second.

Then the second log line console is executed and does its job. After 1 second, the console.log in setTimeout executes, resulting in the above output. This is how asynchronous code works in JavaScript.

In the next section, we will learn about callbacks with asynchronous JavaScript using a fun Github and Twitter example.

Callbacks with Async JavaScript

“Callback” is a word you have heard in real life: you call a friend, but your friend does not pick up the phone. You reach their voicemail and leave them a message to call you back. Your friend hears the message and, let's say two hours later, they call you back. The concept is the same in programming and JavaScript, especially with asynchronous execution.

We won't go into detail about events and how functions can be passed as parameters in Javascript — that's a topic out of the scope of this article. One thing to mention is that callbacks can be used with synchronous code too.

To understand callbacks in the context of asynchronous code, let's see a code example written for the browser. In this snippet, you will fetch the last Tweet from a person by only using their GitHub username.

First, you will call the GitHub API to get the user's details, including their Twitter username. Then, you will call the Nitter RSS feed to get their last tweet.

We won't use the official Twitter API, as this involves authentication and complicates the process. Given our clear goal, we'll use the below code (on JsFiddle for your reference):

const request = window.superagent;
const parser = new RSSParser();
const CORS_PROXY = "https://cors-anywhere.herokuapp.com/";

request
  .get("https://api.github.com/users/abraham")
  .set("Accept", "application/json")
  .end((err, res) => {
    if (!err) {
      console.log("Twitter Username: ", res.body.twitter_username);

      parser.parseURL(
        `${CORS_PROXY}https://nitter.net/${res.body.twitter_username}/rss`,
        (err, feed) => {
          if (!err) {
            console.log(
              `The last tweet by ${res.body.twitter_username} is - ${feed.items[0].title}`
            );
          }
        }
      );
    }
  });

console.log("This would log first");

You are using Superagent and RSS parser to get the task done. Superagent and RSS parser both support callback and the Promises API (coming up next). If this code was written for the backend with Node.js, we wouldn't need the CORS proxy. It is needed for the frontend.

The code starts with instantiating superagent as a request, RSSParser, and CORS proxy. Then the first request is made for our user — abraham — to the GitHub API. On the request end, an anonymous callback is made with err and res.

The Twitter username is plucked out of the results only if there is no error. The RSS parser makes another request to parse the RSS URL and get the tweets. This call results in the execution of another callback.

Here's the output of the above code execution:

This would log first
Twitter Username: abraham
The last tweet by abraham is - She-Hulk and Saul Goodman crossover episode?

If you want to dive a bit deeper into how the V8 engine handles async operation, please watch the amazing talk 'Help I'm stuck in an event loop' by Philip Roberts. He explains the event loop in a clear and concise way.

Next, let's learn about callback hell.

Callback Hell in JavaScript

As seen above, writing callback-oriented code for async operations does not feel natural. On top of that, if three or more callbacks are needed to complete a task, our code becomes difficult to write, understand, and eventually manage.

So when you write JavaScript in a way where execution takes place visually from top to bottom with multiple complex levels of callbacks, you land in callback hell territory. One of the easiest ways to understand callback hell is through visuals:

If you don’t want to get into the callback hell zone, you can use promises with async JavaScript.

Async JavaScript with Promises

Promises (also known as futures in other languages) are objects that represent an eventual completion or failure of an asynchronous task, resulting in a value.

Ok, you can understand it better with another analogy.

Let’s say your friend promises to meet you over the weekend. Then Saturday comes. If the friend actually meets you, the promise is "fulfilled". If your friend does not show up for the meeting, the promise is "rejected". Until Saturday, the promise is "pending".

The same concept can explain any asynchronous operation — for example, calling an external URL/API, or reading a file from a disk.

When an API is called, the promise object is pending until an answer comes back. If all goes well, the promise is fulfilled, and the then method gets the result. In case of failure, the catch method is called, which has the error object. If you are a visual person, this flow chart by MDN web docs can help you to comprehend this idea.

At this juncture, you will convert the above callback code into promises. Superagent and RSS Parser both already provide a promise-based API. The code to call GitHub's API, then get the user abraham's Twitter username (one of the popular GitHub users in the US), and call the Nitter RSS for the username will look like this:

const request = window.superagent;
const parser = new RSSParser();
const CORS_PROXY = "https://cors-anywhere.herokuapp.com/";
let twitterUser = "";

request
  .get("https://api.github.com/users/abraham")
  .set("Accept", "application/json")
  .then((res) => {
    console.log("Twitter Username: ", res.body.twitter_username);
    return res.body.twitter_username;
  })
  .then((twitterUserName) => {
    twitterUser = twitterUserName;
    return parser.parseURL(
      `${CORS_PROXY}https://nitter.net/${twitterUserName}/rss`
    );
  })
  .then((feed) => {
    console.log(`The last tweet by ${twitterUser} is - ${feed.items[0].title}`);
  })
  .catch((err) => {
    console.log(`Error occurred ${err.message}`, err);
  });

console.log("This would log first");

It is mainly the same code from an execution point of view, but written differently. It uses promises in place of callbacks now.

If promises are not supported natively by Superagent and RSS Parser, they can be written with JavaScript’s Promise object too, but you don't have to do it on your own. In Node.js, we can also do it with the util.Promisfy function — a topic for another post.

There are some things you should pay attention to in the above code snippet. First, to make the Twitter username available to other then methods, a twitterUser variable is set before the request.get to an empty string. When the value is available, it is set to the last used Twitter username value, as it is still in scope.

In the case of an error in the then methods, the error is sent to the catch method, not executing the next then method.

You see a promise chain here — the first promise calls GitHub, and the second one gets the latest Tweet from Nitter in the second chain. This can be confusing. We can do this more cleanly with async/await syntax — we'll cover this in the next section.

When the above code is executed, it gives output like the below, which is not different than the callback output:

This would log first
Twitter Username: abraham
The last tweet by abraham is - She-Hulk and Saul Goodman crossover episode?

You can also view the promise code snippet on JSFiddle and play around with it.

If promises look interesting to you, please dig deeper into Promise.all and Promise.race for other ways to run promises concurrently.

Now, let's learn about a modern way of working with promises — using async and await syntax.

Async with Await for Promises

Async with await is the modern way of working with promises in a much cleaner style. Async/await is more syntactic sugar on top of promises than a completely new feature of ECMAScript.

Async allows you to write promise-based code as if it was synchronous. An async function will always return a promise, implicitly making it easier to work with. The await keyword can only be used in async functions and waits for the promise to come to a completed state. A complete state here refers to either a 'fulfilled' or 'rejected' state. There are discussions of top-level await as well, but it has not become mainstream as of yet.

With all of that information, now you will convert the above code with promise, then catch to a more "comfortable" async await version, as follows:

const request = window.superagent;
const parser = new RSSParser();
const CORS_PROXY = "https://cors-anywhere.herokuapp.com/";

async function getLatestTweet(githubUsername) {
  try {
    const gitHubResponse = await request
      .get(`https://api.github.com/users/${githubUsername}`)
      .set("Accept", "application/json");
    console.log("Twitter Username: ", gitHubResponse.body.twitter_username);
    const twitterUsername = gitHubResponse.body.twitter_username;
    const feed = await parser.parseURL(
      `${CORS_PROXY}https://nitter.net/${twitterUsername}/rss`
    );
    console.log(
      `The last tweet by ${twitterUsername} is - ${feed.items[0].title}`
    );
  } catch (err) {
    console.log(`Error occurred ${err.message}`, err);
  }
}

getLatestTweet("abraham");
console.log("This would log first");

The above code is similar to the promise code with .then and .catch. The main difference here is that all the logic is wrapped in an async function called getLatestTweet. The promises are unwrapped with an await which, as the keyword says, waits before going to the next line. As the async operations “behave like” sync code, values dependent on the async task can easily be assigned to variables like the twitterUsername. That is why try/catch also makes more sense here.

In case of any error, as the code acts like sync code, it will be caught in the catch block. For promises using .then and .catch, if the .catch part is missed, errors will get lost. Using async/await makes the code seem synchronous, which is good. Still, overuse of async/await defeats the power of using an async language like JavaScript where "multiple" things can be done simultaneously.

The above code snippet gives out the following output, based on the last tweet by abraham:

This would log first
Twitter Username: abraham
The last tweet by abraham is - She-Hulk and Saul Goodman crossover episode?

The async/await code snippet is also available on Jsfiddle for your reference.

You have learned three ways of handling asynchronous code in JavaScript. In the following section, you will find out how to make your JavaScript async code more efficient.

Make Your JavaScript Async Code More Efficient

You have seen three variations of doing two HTTP calls, one dependent on the other. There are some cases to consider that can make your code more efficient.

Firstly, use async/await wisely. Do not code JavaScript like PHP, using async/await left, right, and center. This will block the event loop.

For example, if you have to call the GitHub API for five usernames, there is no need to call them one by one.

It can be done concurrently with Promise.all. With this approach, be careful that you don't hit the API's rate limit, as calls will be made concurrently. Below is a quick example of Promise.all in action:

const request = window.superagent;

async function getLatestTweets() {
  const usernames = ["bdougie", "abraham", "RamiKrispin"];
  try {
    const requests = usernames.map((username) => {
      return request
        .get(`https://api.github.com/users/${username}`)
        .set("Accept", "application/json");
    });

    const responses = await Promise.all(requests);

    for (const response of responses) {
      const responseData = response.body;
      console.log(
        `Twitter username for GitHub user ${responseData.login} is ${responseData.twitter_username}`
      );
    }
  } catch (err) {
    console.log(`Error occurred ${err.message}`, err);
  }
}

(async () => {
  console.log("Start");
  await getLatestTweets();
  console.log("Done!");
})();

You can check the code on JSFiddle too. When it runs, it gives the following output:

Start
Twitter username for GitHub user bdougie is bdougieYO
Twitter username for GitHub user abraham is abraham
Twitter username for GitHub user RamiKrispin is Rami_Krispin
Done!

The main mantra here is to think of parts of the code that are not dependent on the previous block. Run parts of the code concurrently, so these parts can be broken into smaller functions. Work is made faster, utilizing all available resources like CPU and memory.

Wrap Up

In this post, we covered the difference between synchronous and asynchronous code and execution models. Then we covered three ways to handle async code in JavaScript, using callbacks, promises, and async/await (with an example calling two URLs).

Finally, we saw an example of how to write async code efficiently in Javascript using concurrency, splitting independent code parts into different functions.

Happy coding!

P.S. If you liked this post, subscribe to our JavaScript Sorcery list for a monthly deep dive into more magical JavaScript tips and tricks.

P.P.S. If you need an APM for your Node.js app, go and check out the AppSignal APM for Node.js.

How to outclass your competition in take-home coding challenges

Geshan Manandhar — Thu, 25 Mar 2021 00:00:00 +0000

Like it or not, your hiring process will most of the time involve take-home coding challenges. You can either perform poorly in it or excel. With the steps in this guide, you can surely outclass your competition in any take-home coding challenge. Follow on to find out how:

A take-home coding challenge is a mini project or a small task that your prospective employer gives you to do for assessing your technical skills and knowledge. You are given a task of generally moderate complexity and that can be finished over a weekend or a similar amount of time.

There are other ways to assess your technical skills in place of take-home coding challenges. One of them is a timed coding challenge on HackerRank or similar websites.

In my opinion, another effective alternative to a take-home coding challenge is pair-programming on a task.

Coding side-by-side (or virtually) on the same task can give much better insight into the thought process. It also presents more opportunities to make the process more collaborative and less scary.

You, me, and many other software engineers have done one or more of these take-home coding challenges in their search for a new role or their first role. Let’s look further at how you can outrival your competition:

What is being assessed? #

With a take-home coding challenge the main thing that is being assessed is if you can write code as per the company’s levels and standards. Because you are given ample amounts of time, you are expected to provide a decent quality of output.

On top of it, the decisions you are implying in the form of code is also being evaluated. You are also being analyzed on how well have you followed the instructions.

If a take-home coding challenge clearly instructs you not to use an external library and you still use it. It is pretty sure that you will fail it and not proceed to the next step.

Another aspect implicitly being assessed is your written communication ability. How you write a clear and concise readme will also play a vital role to move you forward in the hiring process. So think about these things before submitting your solution. Make a small checklist of the steps below, that will definitely help you.

Time to go to the steps that will help you outshine your competitor in take-home coding challenges.

Steps to outclass your competition in take-home coding challenges #

Now it is time to know the steps you can take to excel in take-home coding challenges leaving your competition far behind.

Limit the scope and time investment #

It is important to know what to do for a take-home coding challenge. It is crucial to determine what not to do for the challenge. You have to be realistic and practical.

Set aside a definite amount of time to complete the task. Then analyze what can be done in that time limit. Similarly, make a list of things that cannot be done and/or can be done better for your take-home coding challenge.

Limiting the scope gives you the leverage to perform very well in the things you decide to do. This is where you can blow away your competition.

Other candidates will try to do lots of things. This equates to learning multiple things within the deadline of the take-home coding challenges.

That will most likely result in low-quality output or late delivery. This is one of the most important things you must have that early advantage over your competition.

Plan and do step-by-step pull requests #

By this point, you know what to do and how much time you have allotted for the take-home coding challenge. The next logical step is to plan. Break down the whole task into smaller parts. If you can, put an approximate time on each task it would be great. First, solve the problem, then write the code. Break the task to smaller parts to make it easier to plan then execute.

Once the take-home coding challenge is broken down into smaller chunks, start with a basic readme. Then open a pull request for it.

After that, start coding and make sure to commit logically. As soon as things are working make a commit. It also enables coming back if you mess up anything going forward.

Below is an example of the pull requests I made for a coding challenge:

Keep your pull request small. Work as if you are working in a team and review your own pull requests. Step by step PRs and logical commits will help you outdo your competition. This tells the evaluators a clear story of how you reached the solution.

Write automated tests #

Generally speaking, a take-home coding challenge won't specifically ask you to write tests. I would recommend investing a bit more time and write tests.

If you can write automated unit tests first using a TDD approach. If you write tests after code that's fine too. The main point is to have automated tests and know the test's code coverage on the take-home coding challenge.

The focus must be on writing clean and meaningful tests. This will automatically shape up your code to be much more readable and maintainable.

If you sprinkle up some CI/CD goodness on top of tests, it will give you some extra brownie points.

Also explore other forms of automated testing like mutant testing or some integration/end-to-end tests, if you have time. This will help outperform your competition. It will also indicate to the evaluators that you know your craft well.

Dockerize it #

Similar to the above point, using Docker can give you another advantage while doing the take-home coding challenge. Docker's popularity has exploded in the last few years. It has also changed the way we software engineers work.

With docker for the development environment, you can easily use the needed version of the language/framework for the take-home coding challenge.

It will also make it far easier to try and test your solution for the evaluators. If they try to run the solution on their machines you will get a better chance to be advance to the next step.

A dockerized app not only makes it a breeze to deploy, it also gives you an edge as some of your competition will not bother to use it. You could easily deploy your web application to say Google Cloud Run if you dockerized it. With a working URL, your final output will have more chances of being checked and commented on by the evaluators.

Write a killer readme #

Last but not least, having a stellar readme makes a big difference. As software engineers, we are usually good at our technical skills. Conversely, we don't generally excel in communication. This is where writing a clear and well-structured readme plays a crucial role.

You will want to know how to structure a readme, below is part of a readme I wrote up for a take-home coding challenge.

The readme format below is tried and tested as I was called in for an interview. This worked very well.

You can read the full readme to get an idea of what to write in a readme, how to structure it, and how to write it. I think the “Decisions and Tradeoffs” section was one of the most important sections of the readme.

Some other things to be very mindful of:

Don’t make the evaluator think and search, make the main parts of the coding challenge obvious and linked in the readme.
Explain how you came to the solution, brief your thought process
Be clear of the scope you covered, as you are not expected to do everything in the given amount of time
Always mention decisions and tradeoffs you took to complete the take-home coding challenge
Also, be clear about things that could have been done better, but it was not done due to reasons like time availability.

Even as an evaluator of take-home coding challenges, I have seen many candidates write up a poor readme or they don’t include it at all. It just decreases your chances of getting a fair review. That results in friction for you moving forward to the next step of the hiring process.

Bottom line: write a killer readme and blow your competition away :).

Conclusion #

Needless to say, you absolutely must read the take-home coding challenge description at least twice. This will help you understand the problem space and what the evaluators are expecting from you.

Another ultra-important thing is don’t mess up the basics. For instance, name your variables appropriate to the context don’t name then a, b or i, and j. The same goes for your function names.

I have been on both sides of the table, a candidate as well as an evaluator of take-home coding challenges. You will need to balance things well so that your application moves forward smoothly.

“A user interface is like a joke. If you have to explain it, it’s not that good”. — Martin Leblanc

Without doubts, the same things can be said for code, if you need to explain your code to someone you are not making it simple enough. This becomes paramount as the evaluator will check the code without you on the side so be mindful of this too.

I hope these steps and tips will help you ace your next take-home coding challenge, best of luck!

3 Software Engineering podcasts you must subscribe to now

Geshan Manandhar — Fri, 29 Jan 2021 04:22:22 +0000

Software Engineering podcasts have been my companion for a long time. I have listened to software engineer podcasts for more than a decade. Software Engineering podcasts give you the latest news and views in a great format that is equally easy to consume too. You can also read the previous parts 1, 2, 3, and 4. Here are the 3 podcasts all software engineers should subscribe to now:

Command Line Heroes (from Redhat) by Saron Yitbarek

Command line heroes is a gem of a podcast. The topics are well researched and in-depth. It covers a wide range of topics that are very relevant to all software engineers. I remember listening to the episode about serverless from Season 2. It was a breath of fresh air having comments from Saron, bytes from the archive, and great add-ons by multiple guests including Andrea Passwater.

This award-winning podcast defines itself as:

Command Line Heroes tells the epic true tales of how developers, programmers, hackers, geeks, and open source rebels are revolutionizing the technology landscape.

Amongst other seasons, my favorite was Season 3. There first episode about Python was simply amazing and enlightening. I was happy to know about the Python programming language’s benevolent dictator for life: Guido van Rossum who created this multipurpose language.

Saron has other podcasts too like Coding Newbie which is great too. Command line heroes’ episodes are not that long but the way the music, commentary, and other things are blended together show the mark of a great podcaster. The starting generally hooks you in to listen to the whole episode.

Developing Up By Mike Miles

Developing up is an amazing software engineering podcast. The good aspect of this podcast is it focused on the non-technical side of our careers. I think having great soft skills is crucial to do a software engineering job well as well as indispensable to climb up the career ladder. I remember the episode pretty well where Mike and Karl talk about public speaking. It was definitely a great one among others.

Developing up describes itself as:

A podcast focused on the non-technical side of being a developer because your career is about more than the code you write.

With 50 episodes since 2016, listening to this software engineering podcast you will surely learn some needed non-technical skills. Ranging from Pair programming and code reviews to working remotely and imposter syndrome.

The episodes are relatively short and Mike does a great job of asking amazing questions. This brings out insightful and relatable information. I would really recommend you to subscribe to this software engineering podcast.

The SAAS Podcast by Umer Khan

As usual, for the third software engineering podcast, I have something that is more related to the business side fo things. As software engineers, if we understand the domain and why we are doing things this way it makes our work much more meaningful.

One episode I remember pretty well is the chat between Dave and Umer about ClickFunnels. Dave describes the ClickFunnels journey and how it made millions of dollars.

The SAAS podcast defines itself as:

Over 250 in-depth interviews with proven SaaS founders and entrepreneurs. Get actionable insights to help you build, grow, and scale your SaaS business

There are many other episodes I like from THE SAAS software engineering podcast, like the one where Krish talks about scaling a SAAS business. Omer does a great job of researching and interviewing the host. The episodes are a bit long but worth the time.

Conclusion

Even though due to no commuting to work for the past ~6 months I have listed to podcasts a lot less. I hope you enjoy listening to the above software engineering podcasts. Happy listening.

Node.js Postgresql tutorial: Build a simple REST API with Express step-by-step

Geshan Manandhar — Thu, 14 Jan 2021 11:12:23 +0000

Node.js can be used efficiently with relational databases like PostgreSQL. In this post about Node.js PostgreSQL tutorial, we are going to build a REST API for Quotes step-by-step using Express Js.

Prerequisites
Steps for Node.js PostgreSQL tutorial
1. Setup Express with express generator
  1. Delete the public folder
  2. Delete unnecessary existing routes then create a new route for quotes
  3. Change index route to give out JSON
2. Setup PostgreSQL with quote table
3. Link Node.js with Postgres
4. Show Quotes - GET API with pagination
5. Save a new quote - POST API for Node.js PostgreSQL tutorial
  1. Adding validation for creating quotes POST API
TLDR; quick rundown
Conclusion

You can read my previous tutorial to try Node.js with MySQL. You should read this guide to use Docker with Node.js, it is also a step-by-step guide. You can, of course, carry on with this Node.js with PostgreSQL tutorial :).

For your convenience, each step has been carved out as a distinct pull request so that you can follow the tutorial with ease.

Prerequisites

You have Node.js (preferably Node 14.x) installed and running on your machine (or Node.js running with a docker container).
You are aware of how Node.js generally works and also have a bit of Express Js experience.
Having some knowledge of Git and GitHub will be really useful.
For the database we will use a free database on ElephantSQL, so please register and set up a free PostgreSQL database there. Of course, you should know how a relational database works.
You are able to code using an IDE. I will be using VS Code as an editor but you are free to use any code editor of your choice for this Node.js PostgreSQL tutorial.

Steps for Node.js PostgreSQL tutorial

We will be building a very simple REST API with Express Js that can send out some quotes. A quick refresher on what REST APIs are would be greatly helpful at this point.

It will be great to read about HTTP verbs and brush up on some cURL commands too. We will be using cURL to run the examples.

At this juncture, we believe that your Node.js is running fine. So let’s start with setting up Express js:

Setup Express with express generator

To step Express js with express-generator run the following command:

npx express-generator --no-view --git nodejs-postgresql

The --no-view parameter tells the generator to generate the Express app without any view like Pug. The other --git parameter indicates that we want to add the default .gitignore file in our Express app.
It will create the needed files in nodesj-postgresql directory. Your express will be set up when the command runs successfully. To quickly check if Express is setup correctly run the below:

cd nodejs-posgresql && npm install && DEBUG=nodejs-posgresql:* npm start

You should see something similar on your browser when you hit http://localhost:3000 on it:

The generated barebones Express Js app can be viewed in this pull request.

Delete the public folder

Because we are building a REST API for quotes for this Node.js PostgreSQL tutorial we don’t need any CSS or JS. Therefore, we will delete the generated public folder as we will deal with JSON.

To delete the generated public folder execute the following on your terminal:

rm -rf public

Delete unnecessary existing routes then create a new route for quotes

At this juncture, we will delete the unnecessary users' route found in routes/users.js. Consequently, we will add routes/quotes.js file that will have the following content:

const express = require('express');
const router = express.Router();

/* GET quotes listing. */
router.get('/', function(req, res, next) {
  res.json({
    data: [
      {
        quote: 'First, solve the problem. Then, write the code.',
        author: 'John Johnson'
      }
    ],
    meta: {
      page: 1
    }
  });
});

module.exports = router;

For now, it will give a static output of only 1 quote as shown above. We will link up the quotes route in the app.js file like below:

var express = require('express');
var path = require('path');
var cookieParser = require('cookie-parser');
var logger = require('morgan');

var indexRouter = require('./routes/index');
var quotesRouter = require('./routes/quotes');

var app = express();

app.use(logger('dev'));
app.use(express.json());
app.use(express.urlencoded({ extended: false }));
app.use(cookieParser());
app.use(express.static(path.join(__dirname, 'public')));

app.use('/', indexRouter);
app.use('/quotes', quotesRouter);

module.exports = app;

The changes on the above file are only on lines 7 and 18 where the users’ router has been replaced with the quotes one.

Change index route to give out JSON

The last change in this step is on line 6 of index.js file found in the root of the project. We will edit it to send out JSON in place of rendering a view. The file will look like before after this change:

var express = require('express');
var router = express.Router();

/* GET home page. */
router.get('/', function(req, res, next) {
  res.json({message: 'alive'});
});

module.exports = router;

You can check all the changes made in this step in this pull request.

To swiftly view the output of the above changes run the following:

DEBUG=nodejs-postgresql:* npm start

Then hit http://localhost:3000/quotes on the browser tab, you will see something like below:

We will fetch the quotes from our PostgreSQL database in the next step.

Setup PostgreSQL with quote table

We want to save you from the hassle of creating and maintaining a database locally on your machine. You can have a free PostgreSQL database on Elephant SQL with 20 MB data and 5 concurrent connections. It is more than enough for the purpose of this tutorial.

To create the PostgreSQL database on Elephant SQL after registering please follow this documentation. Create the free (tiny turtle) database in the data center of your choice. If you want to learn more about PostgreSQL follow this multi-part tutorial on Elephant SQL.

After that, to create the quote table run the following SQL in the “browser” section of the created database:

CREATE SEQUENCE quote_id_seq START WITH 1 INCREMENT BY 1 NO MINVALUE NO MAXVALUE CACHE 1;

CREATE TABLE quote (
    id bigint DEFAULT nextval('quote_id_seq'::regclass) NOT NULL PRIMARY KEY,
    quote character varying(255) NOT NULL UNIQUE,
    author character varying(255) NOT NULL,
    created_at timestamp with time zone DEFAULT CURRENT_TIMESTAMP NOT NULL,
    updated_at timestamp with time zone DEFAULT CURRENT_TIMESTAMP NOT NULL
);

Here is how it looks on the Elephant SQL interface:

This is a very simple table with 5 columns. The first one is the id which is a sequence and primary key. Then there are quote and author both are variable characters.

After that created_at and updated_at are both time stamps. There is a unique index added to the quote column so that we don’t have the same quote more than once. After the table is created we will fill up some quotes in the quote table executing the insert SQL below:

INSERT INTO quote (quote, author) VALUES 
('There are only two kinds of languages: the ones people complain about and the ones nobody uses.', 'Bjarne Stroustrup'), 
('Any fool can write code that a computer can understand. Good programmers write code that humans can understand.', 'Martin Fowler'), 
('First, solve the problem. Then, write the code.', 'John Johnson'), 
('Java is to JavaScript what car is to Carpet.', 'Chris Heilmann'), 
('Always code as if the guy who ends up maintaining your code will be a violent psychopath who knows where you live.', 'John Woods'), 
('I''m not a great programmer; I''m just a good programmer with great habits.', 'Kent Beck'), 
('Truth can only be found in one place: the code.', 'Robert C. Martin'), 
('If you have to spend effort looking at a fragment of code and figuring out what it''s doing, then you should extract it into a function and name the function after the "what".', 'Martin Fowler'), 
('The real problem is that programmers have spent far too much time worrying about efficiency in the wrong places and at the wrong times; premature optimization is the root of all evil (or at least most of it) in programming.', 'Donald Knuth'), 
('SQL, Lisp, and Haskell are the only programming languages that I’ve seen where one spends more time thinking than typing.', 'Philip Greenspun'), 
('Deleted code is debugged code.', 'Jeff Sickel'), 
('There are two ways of constructing a software design: One way is to make it so simple that there are obviously no deficiencies and the other way is to make it so complicated that there are no obvious deficiencies.', 'C.A.R. Hoare'), 
('Simplicity is prerequisite for reliability.', 'Edsger W. Dijkstra'), 
('There are only two hard things in Computer Science: cache invalidation and naming things.', 'Phil Karlton'), 
('Measuring programming progress by lines of code is like measuring aircraft building progress by weight.', 'Bill Gates'), 
('Controlling complexity is the essence of computer programming.', 'Brian Kernighan'),
('The only way to learn a new programming language is by writing programs in it.', 'Dennis Ritchie');

After you have inserted the 17 rows, if you run the following on the Elephant SQL browser interface:

SELECT * FROM quote;

You should see something similar to below:

You can find the init database SQL file in this pull request. As our database is set up and ready let’s proceed to link it up with the Node.js Express application.

Link Node.js with Postgres

To link up the Node.js Express Js application with the database that we have set up we will need to install the Postgres npm library. To get this useful library please run the following command:

npm install --save pg

The changes we have got by installing this npm package are in this pull request. Time to proceed to add the GET quotes API route.

Show Quotes - GET API with pagination

When you go to http://localhost:3000/quotes after starting the Express Js app, you can see something like below:

{
  "data":[
    {
      "quote":"First, solve the problem. Then, write the code.",
      "author":"John Johnson"
    }
  ],
  "meta":{
    "page":1
  }
}

Now we will replace this by fetching data from the PostgreSQL database on Elephant SQL. To do this, we will need to connect to the database.

Let’s create a config.js file on the root level. This config file has the database credentials and other configs like below:

const env = process.env;

const config = {
  db: { /* do not put password or any sensitive info here, done only for demo */
    host: env.DB_HOST || 'otto.db.elephantsql.com',
    port: env.DB_PORT || '5432',
    user: env.DB_USER || 'cklijfef',
    password: env.DB_PASSWORD || 'V1qidES5k3DSJICDRgXtyT8qeu2SPCZp',
    database: env.DB_NAME || 'cklijfef',
  },
  listPerPage: env.LIST_PER_PAGE || 10,
};

module.exports = config;

Subsequently, we will need to add a services/db.js file that will use a pool to run our SQL queries. It will look like below:

const { Pool } = require('pg');
const config = require('../config');
const pool = new Pool(config.db);

/**
 * Query the database using the pool
 * @param {*} query 
 * @param {*} params 
 * 
 * @see https://node-postgres.com/features/pooling#single-query
 */
async function query(query, params) {
    const {rows, fields} = await pool.query(query, params);

    return rows;
}

module.exports = {
  query
}

After that we will add a helper.js file on the root level that will help us format the results and calculate the offset for pagination. It will have the following contents:

function getOffset(currentPage = 1, listPerPage) {
  return (currentPage - 1) * [listPerPage];
}

function emptyOrRows(rows) {
  if (!rows) {
    return [];
  }
  return rows;
}

module.exports = {
  getOffset,
  emptyOrRows
}

At this juncture, we will add services/quotes.js file which will have contents as below:

const db = require('./db');
const helper = require('../helper');
const config = require('../config');

async function getMultiple(page = 1) {
  const offset = helper.getOffset(page, config.listPerPage);
  const rows = await db.query(
    'SELECT id, quote, author FROM quote OFFSET $1 LIMIT $2', 
    [offset, config.listPerPage]
  );
  const data = helper.emptyOrRows(rows);
  const meta = {page};

  return {
    data,
    meta
  }
}

module.exports = {
  getMultiple
}

All of this is glued from the routes file at routes/quotes.js which after the change looks like:

const express = require('express');
const router = express.Router();
const quotes = require('../services/quotes');

/* GET quotes listing. */
router.get('/', async function(req, res, next) {
  try {
    res.json(await quotes.getMultiple(req.query.page));
  } catch (err) {
    console.error(`Error while getting quotes `, err.message);
    next(err);
  }
});

module.exports = router;

The main change here in the routes/quotes.js file is the addition of quotes service. After that, the /quotes is getting the quotes dynamically using the added quotes service.

Please take note that the quotes are already paginated, meaning https://localhost:3000/quotes?page=2 will give out quotes 11-20 as it has 10 quotes per page in the config.js file. The output at this point for page 2 should look something like below:

Now let’s proceed to add the POST quote API which will insert a new quote on the database. As above, you can view all file changes for this step in this neatly organized pull request.

Save a new quote - POST API for Node.js PostgreSQL tutorial

To create a new quote we will keep it a simple POST API. We will not use any validation library and keep the response codes as simple as possible.

The first thing we will do for adding the save new quote endpoint is to add it to the /routes/quotes.js file just above module.exports = router line as shown below:

/* POST quotes */
router.post('/', async function(req, res, next) {
  try {
    res.json(await quotes.create(req.body));
  } catch (err) {
    console.error(`Error while posting quotes `, err.message);
    next(err);
  }
});

module.exports = router;

For the time being, we will not add any code level validation. The database table quote has the quote field required and 255 characters. So, if the quote is empty it will get a database level error. Unlike MySQL, PostgreSQL will give an error if the quote is longer than 255 characters.

In a more real-life scenario, I would recommend using a validation library for these kinds of cases. For now, let’s add the create method in /services/quotes.js like below:

async function create(quote) {
  const result = await db.query(
    'INSERT INTO quote(quote, author) VALUES ($1, $2) RETURNING *',
    [quote.quote, quote.author]
  );
  let message = 'Error in creating quote';

  if (result.length) {
    message = 'Quote created successfully';
  }

  return {message};
}

module.exports = {
  getMultiple,
  create
}

After you run the server, you can try the following curl to see if it create a new quote:

curl -i -X POST -H 'Accept: application/json' \
    -H 'Content-type: application/json' http://localhost:3000/quotes \
    --data '{"quote":"Before software can be reusable it first has to be usable2.","author":"Ralph Johnson"}'

It should come back with a 200 response saying the quote has been created. You can try other quotes from this post.

With the create quote POST API working. You can easily build upon it to create the edit and the delete quote endpoints with respective UPDATE and DELETE SQL statements. Just be careful to pass the right id to carry out those operations.

Similar to the previous steps, all the code changes for this step can be found in this pull request.

Adding validation for creating quotes POST API

Till now it should be functional but we should not push validation to the database layer as it will be more resource expensive. So in the next part of this step, we will add validation on the code level.

We will add a validateCreate method above create method in /services/quotes.js to do the validation like below:

function validateCreate(quote) {
  let messages = [];

  console.log(quote);

  if (!quote) {
    messages.push('No object is provided');
  }

  if (!quote.quote) {
    messages.push('Quote is empty');
  }

  if (!quote.author) {
    messages.push('Quote is empty');
  }

  if (quote.quote && quote.quote.length > 255) {
    messages.push('Quote cannot be longer than 255 characters');
  }

  if (quote.author && quote.author.length > 255) {
    messages.push('Author name cannot be longer than 255 characters');
  }

  if (messages.length) {
    let error = new Error(messages.join());
    error.statusCode = 400;

    throw error;
  }
}

async function create(quote){
  validateCreate(quote);

  const result = await db.query(
    'INSERT INTO quote(quote, author) VALUES ($1, $2) RETURNING *',
    [quote.quote, quote.author]
  );
  let message = 'Error in creating quote';

  if (result.length) {
    message = 'Quote created successfully';
  }

  return {message};
}

So now if you try the below cURL without an author, when the server is running it will show an error:

curl -i -X POST -H 'Accept: application/json' \
    -H 'Content-type: application/json' http://localhost:3000/quotes \
    --data '{"quote":"Before software can be reusable it first has to be usable."}'

It will show something like below:

These changes are also reflected in a pull request.

On dev, I would highly recommend using Nodemon as it will restart the server on every file change.

After you have nodemon installed globally you can run the app with the following command:

DEBUG=nodejs-postgresql:* nodemon bin/www

Nodemon is great for development.

TLDR; quick rundown

All the code shown above is in a public GitHub repository, to quickly get started with what has already been built you can follow the steps below:

Clone the repository with: git clone git@github.com:geshan/nodejs-posgresql.git
Then run cd nodejs-postgresql
Subsequently, execute: npm install && npm start
After that, hit: https://localhost:3000/quotes on your favorite browser
You should see the following on your browser:

Conclusion

Setting up a REST API with Node.js and PostgreSQL was pretty fun till now. Still, it is more like scratching the surface. You can use the GitHub repo as a boilerplate to create simple REST APIs.

This tutorial is a great starting point to build a full-on real-life REST API. I hope you can use this as the commencement of something exciting. Best of luck!

How to make PHPUnit Code Coverage 2+ times faster with Pcov compared to Xdebug

Geshan Manandhar — Tue, 05 Jan 2021 11:49:16 +0000

PHPUnit is the de-facto testing library for PHP. With the use of PCov you can speed up PHPUnit code coverage by 2-5 times for PHP 7.0+ application. In this post, we will compare the results of an experiment I did on Laravel framework tests. The tests were run without coverage, then with Xdebug coverage, and finally with PCov all on Github actions. Pcov took half the time to run the PHPUnit tests with code coverage compared to Xdebug, let’s go to the numbers.

Save time on CI builds

What does saving say 12 seconds on each build mean on the long run? If you can save just 12 seconds on each build, it equates to 1 minute in just 5 builds.

In just 100 builds you save 20 minutes and in 1000 builds that number becomes 3 hours and 20 mins.

Think of it as how much it will save in terms of waiting time for your colleagues too.

PHP code coverage with Pcov, not Xdebug

XDebug is a debugger that can do coverage too. PHPdbg is another alternative to Xdebug. Pcov is built for PHPUnit code coverage, not something else.

Michael Dyrynda has also talked about this issue, where he mentions:

You have access to the same output formats that are available to PHPUnit (formatted output, clover, JSON, HTML, etc.) with none of the overhead.

He has also pointed out other issues with Xdebug like hitting the max_nesting_limit too, you should read his blog post too.

Considerations for code coverage with PCov

Pcov is a self-contained PHP code coverage driver for PHP 7 and above
PHPUnit 8 and above supports PCOV out of the box, for PHPUnit 7 and lower you will need pcov-clobber
Pcov makes perfect sense in a Continuous Integration (CI) environment as you don't debug code on a CI :)

To show a real-life scenario we are going to see how long Laravel Framework’s 5700+ tests with 15500+ assertions are going to take in our quick experiment.

Let’s get cracking!

Procedure for faster PHPUnit code coverage

I picked up the Laravel framework not only because it is very popular but also because there were a lot of tests, more than 5700 of them. On top of it, for the Laravel 8.x branch, the tests are running on Github Actions.

Tests for Laravel 8.x run on multiple versions of PHP like 7.3, 7.4,8 on lowest to stable variants. The same tests also run on windows.

Another reason to choose Laravel 8.x was it is using PHPUnit 9.3 which does not need pcov-clobber to get the PHPUnit coverage.

I had blogged about getting started with Unit testing in Laravel in the past which should be a good unit testing refresher. Data provider for PHPunit is also a great way to write less test code but achieve more code coverage.

Below are the steps I took to find out how fast Pcov was against Xdebug for PHPUnit code coverage.

Fork Laravel/framework repo and run tests only for PHP 7.4

To keep things simple, I forked the Laravel/framework Github repo. After that, I change the Github Actions tests workflow to run tests only on PHP 7.4 which the current stable version. You can see the changes I made in this pull request.

Opening the pull requests ran the tests without PHPUnit code coverage and it took 33 seconds to run the tests consuming 257MB of memory.

You can view the details of that test run in this Gitub Actions page, below is a quick screenshot.

Run PHPUnit code coverage with XDebug

I merged the above pull request to run tests only for PHP 7.4. Then I made changes to run the PHPUnit tests with code coverage using Xdebug as the driver. The change is very easy as Gitub action was using shivammathur/setup-php@v2 action. After a bit of Googling, I found that that action had Code Coverage support and it was very easy to enable.

I had to change the coverage from none to xdebug and add --coverage-text to the PHPUnit command making it:

vendor/bin/phpunit --verbose --coverage-text

I made those changes in 2 places in the tests.yml file and that resulted in this pull request. A new pull request = the tests running again in Github Actions CI.

With Xdebug code coverage I did a couple more runs to see if the time taken to run the test vary by much. It was generally the same.

In one of the runs of code coverage with Xdebug took 2 mins 34 seconds and consumed 395 MB of memory.

I am only checking the time for the Execute tests task. You can view the screeshot below:

The code coverage was reported as below, with 75.65% of the lines covered and 68.90% of the methods covered by PHPUnit Code coverage using Xdebug.

PHPUnit Code Coverage with Pcov is 2x faster

Now with the time of 154 seconds for Xdebug, I wanted to see how long it would take the new coverage driver Pcov. To find this out, I again followed a similar approach, went to the Laravel 8.x branch, and started editing the .github/workflows/tests.yml file. I change the coverage from none to pcov thankfully the PHP action supports pcov.

The changes I made are in this pull request. This triggered another build on Github Actions.

This time surprisingly it took just 1 minute 17 seconds and consumed 393 MB of memory.

Again this is for the Execute tests task as seen below:

Same as Xdebug the PHPUnit Code Coverage was reported as 75.65% of the lines and 68.90% of the methods covered by PCov. You can see other test runs in the Actions tab of my Laravel Framework fork.

Quick comparison of Code coverage

Let’s take a quick look at how long the PHPUnit test took with and without code coverage:

PHPUnit Test Run (PHP 7.4 Linux)	Time Taken	Memory Consumed
No Coverage	33 seconds	257 MB
Coverage with XDebug	2 minutes 34 seconds (154 seconds)	395 MB
Coverage with Pcov	1 minute 17 seconds (77 seconds)	393 MB

It is very clear that Pcov took half the time as Xdebug and even consumed lesser memory. PHPUnit code coverage with Pcov took 77 seconds and with Xdebug took double of that at 154 seconds.

In my local run in a docker container, the results were pretty different. For Xdebug these Laravel framework tests took 15 minutes 15 seconds (403 MB memory) and the with Pcov driver the same tests took 3 minutes 25 seconds (399 MB memory).

Pcov was 4.43 times faster on my local machine inside a docker container.

Not only me, but Swashata Ghosh has also reported a 5 times faster code coverage with Pcov in place of Xdebug. Speaking of pure numbers, I ran a check on a test where it took 17 seconds with Xdebug and it took just 1 second with PCov. It was 17 fold faster but that should not be a yardstick to compare Xdebug and Pcov for code coverage.

I did not try PHPDbg as an option because it was not available in the PHP Github action. If you want to quickly switch between Xdebug and Pcov please read this guide.

Conclusion

PHPUnit code coverage is usually coupled with Xdebug. It has a problem the code converge reports are slow with XDebug. PCov is purpose-built for PHPUnit code coverage not debugging and it makes gathering code coverage a lot faster.

If you want to speed up your code coverage and save time on your CI builds use Pcov in place of Xdebug. You will surely like the time saved after the process is done. Happy faster testing and coverage reports!

Working as a software engineer on internal products Vs a customer-facing one in E-commerce companies

Geshan Manandhar — Tue, 05 Jan 2021 05:36:12 +0000

Working on software systems is usually a complex process if not a complicated one. In this post, I am going to highlight the key differences between working for internal software and customer-facing applications with examples related to E-commerce. I have worked with both internal products and customer-facing applications in the past 8 and a half years. Without further ado, let’s get started.

Why e-commerce examples?

E-commerce has been around for decades now. I have worked for fashion e-commerce companies for the past 8 years now on 2 continents. Unsurprisingly, due to COVID-19, many brick and mortar businesses have been forced to jump on the e-commerce bandwagon. This has created many new opportunities. Smaller companies could live with services like Shopify or one of its competitors like Wix. This would result in no or minimal software development.

Businesses with an appetite for in-house software development can even start with something like woo-commerce, Magento, Odoo, or similar. But, at a scale of millions of customers, cookie-cutter generic solutions might not be the best fit.

Either it is too much work to get things customized well or hammering the screw is not a path you want to choose for your business. Therefore, having custom-built solutions that can scale well with business demands is crucial for commercial success.

Assumptions

Having worked for fashion e-commerce companies with lots of internal systems, I assume that many e-commerce companies work the same way.
Examples and experience will be based on physical products especially related to fashion.

I had written about working as a software engineer for a dev shop/agency vs a product company in the past, it should be a good read too. Let’s look at what are internal products and customer-facing applications.

What is an internal product?

Simply put, an internal product is a product used only by the employees of a company. In the case of e-commerce, any software system that is used to do back-office operations will qualify as an internal product.

Some of the examples of internal products include order management systems, picking and/or packing applications. software systems to book parcels with couriers, software to manage returns, etc are also internal applications.

What you see from the outside world as a customer is just the tip of the iceberg. There will be many software systems used internally by e-commerce companies to make your purchase experience better.

Even before a product (SKU) is visible on the catalog pages, it would have passed through multiple internal systems. For instance, it would be uniquely identifiable on the Enterprise Resource Planning (ERP) software. If there is a buying software in place, that SKU would be visible and accessible there too.

In case there is a production and/or studio management system, that same SKU will exist there as well for the imagery and text contents of the product.

After the purchase, there will be a suite of internal applications to make sure the product reaches the customer easily. From systems that send the order to the EPR, software that talks with couriers to even systems that track customer’s returns.

There will be a web of internal products not directly visible to the end customer. On the other hand, there will be some software systems that are available for use to end customers, let's learn what they are.

What is a customer-facing application?

Any application used by the end customer can be referred to as a customer-facing application. First and foremost, the website customers use to browse through the catalog and buy is an important customer-facing application.

Another must-have customer-facing application these days are native mobile apps that serve a similar purpose as the website. By now, it is clear anyone browsing or buying from the e-commerce website is a customer or lead for the website.

Depending on how the e-commerce company breaks down its software services there can be many customer-facing applications.

For instance, the wishlist can be a standalone application on its own. The cart can be another application that works in tandem with the other related customer-facing parts. The catalog can be a big independent application on its own with the search capabilities inside it. Checkout, the money earner can also be another software application.

Any application the end customer has direct access to qualifies as a customer-facing application. Ranging from the catalog page to the return page used by the customer to request a return.

Customer-facing applications will work closely with internal applications to provide superior service to the end customer.

Time to be informed of the difference between working as a software engineer on these two sides of the spectrum.

The differences in working on internal applications vs customer-facing ones

Now let’s dig more into the differences between working as a software engineer on internal applications and customer-facing ones.

Fewer performance and scalability considerations for internal applications

As software engineers, we always have to be conscious of the performance impact of our architecture and code decisions. These performance impacts have a higher value for customer-facing applications compared to internal ones. An internal application being 100 milliseconds slower won't have a big impact. On the other hand, if say the checkout is 100 milliseconds slower, that would be a big concern.

Of course, this in no way promotes writing poor performing code for internal applications. Still, even if you would want to do the refactoring of code, performance upgrades in 1-2 weeks from now it will not hurt the business much.

Another aspect is the load the application will receive. For internal products, it will be predictable as the number of users or usage patterns will be clear. For an open to the whole world customer-facing application, you could potentially get millions of users in a day. This means those millisecond/microsecond optimizations will need some careful consideration for customer-facing applications compared to internal products.

Similarly, the scalability of the application also comes into play at this point. Customer-facing applications will be provisioned more resources to make them more scalable as the load is difficult to predict. Usually, internal applications run on much fewer resources as they don’t get hit millions of times a day.

Customer-facing applications are more security sensitive

Depending on your customer base, a customer-facing application can be accessed by anyone in the world. Conversely, an internal product's users are limited. Because of this, access to the application can be behind a VPN/firewall or even allowed a group of IPs. With all of this network-level filtering in place, the application-level security can be a bit relaxed for internal products.

On the other hand, as a software engineer, you will have to be on your toes for customer-facing applications. Even though the attack vector may be similar but the people who could try to attack is a lot more.

This is where knowledge of OWASP top 10 vulnerabilities will come in very handy for you.

Security is everyone’s responsibility. In no means or form, I am suggesting to write less secure code for internal applications. The only thing is the cost and blast radius of a security breach of an internal product will be lesser than a customer-facing one. This is surely subject to people’s intentions if an employee does a security breach on a customer database from an internal product that is a completely different scenario to process.

Minimal use of feature flags, A/B testing, and gradual rollouts on internal applications

As per my experience, I have never seen an internal application being A/B tested. The reason is simple: you can just go to the user and have a chat to come to a better decision. As the users are limited and within the same organization you can even ask the user to use a certain browser or software. Once, we deployed a go executable on like 10 photographers’ Mac to be able to upload images to S3 much faster. There was no need for an A/B test there. Another reason not to do A/B testing for internal products is the limited scale which will result in skewed results due to low volumes.

There is a similar situation for using feature flags. It will be needed now and then but not all the time. I have also rarely witnessed gradual rollout like 1% of the users for the first week, then 5% next week and slowly going to 100% on internal products.

On the contrary, for customer-facing applications especially for e-commerce A/B testing is like an agreed norm. Many times one user will see the checkout button “green” but another sure will see it “blue”.

Similarly, feature flags are used a lot more for customer-facing applications. In my own experience, when I first added PayPal to our website back in 2012 it was only available for 2 emails initially, then it went live for all company emails. Only after a couple of days of testing, we released it to everyone.

I share a similar experience with gradual rollouts on customer-facing applications. One instance I remember is we changed our payment gateway. After a lot of rigorous testing on staging, the first time it was released to production was for just 1% of the customers. Gradually next week it came to 5%. After some fixes, it went to 10, and to reach 100% it took like a month or so.

In terms of measuring impact, internal products are easier. Internal products work on one of these two objectives. Either enable to do a thing or optimize (like speed) of something by X%. On the other hand, customer products have multiple complex metrics. The metrics can range from NPS, conversion rate to return rate, etc. While one metric is improved another metric might be hampered along the way.

Customer-facing applications get more UX love

Customers are the money payers and they should “enjoy” using our websites and mobile apps. Contrastingly, employees who use the internal product, it is part of their jobs. Internal products are built to make their lives easier so UX love is somewhat missing on them.

This certainly does not equate to that all internal products must lack the UX factor. I am just sharing my personal experience from the past 8-9 years of working in e-commerce. Once we built an internal application that was built on bootstrap. I am not saying bootstrap CSS is bad in any way.

My point here is that in that project we didn’t even have a dedicated UX person for designs. The frontend engineers pieced together the app and it was ready to go. Of course, it made the user’s life much easier and their work much faster. Still, some UX love would have been the cherry on the cake.

On the flip side, customer products usually have a dedicated UX person. This person does research, thinks through how the customer will use this new feature. S/he probably even interviews some customers to get the feedback and make that app/feature a UX gem. Making that button round or square is a decision. Showing the message on top or beside the product is another factor. I won’t be wrong if I say customer-facing applications get much more UX attention and work than internal products and honestly it is deserved too.

Another reason internal products don't get much of the UX love is "personas". While customer-facing apps cater to different personas with varied needs, internal products have usually one persona trying to get their job done.

Summary

Here is the summary of the above differences as a simple table:

Criteria	Internal Products	Customer-facing applications
Performance and scalability	A bit of slow performance is Ok.	Application needs to be performant and scaleable under high loads.
Security sensitivity	Not very sensitive as internal applications are being a VPN/firewall.	Highly sensitive to security issues as anyone in the world can access a customer-facing application.
A/B Testing, feature flags, and gradual rollout	These are not used much for internal products.	These are used heavily on customer-facing applications with segmenting too.
UX Love	Internal products don't get much of the UX love.	Customer-facing applications get a lot of UX love :).

Conclusion

The main thing to comprehend is both internal products and customer-facing applications make the overall customer experience better. Don’t focus on only one part, having one of the best checkout experiences but delivering the item after 30 days does not make sense.

Strengthen both sides of the coin. Internal products and customer-facing applications are like two wheels of a chariot. If one fails the chariot can only go in circles leading to nowhere. I hope as a software engineer, you get to work on amazing products on both sides with world-class software engineering practices in place.

Docker build example: how to go from slow to fast docker builds

Geshan Manandhar — Mon, 28 Dec 2020 05:11:51 +0000

In this post, we will see a docker build example of a node js API application starting from slow and ending up in a ~10x faster build. I have already talked about the reasons to use docker for development environment. I have also mentioned how docker changed the way we software engineers work and multi-stage docker build in past posts. For this one let’s focus on the docker build example with a faster build in mind.

Information before jumping in

Familiarity with Docker and the docker build process is required
All examples are based on Docker version 19.03.13, build 4484c46d9d on a Mac
The Currency API app is used for this docker build example

Why faster docker build

There are many reasons you would want your Docker containers to build faster, here are some pressing ones:

It will save the software engineer’s time while waiting for container images to build in the CI/CD pipeline. Imagine this if all your docker build took half the time it would result in a lot less waiting.
It will also save engineers time to build and run the software locally. In this era of microservices if those images would build faster it would help a lot.
The faster build also enables faster deployment and releases. If you wanted to rollback a buggy deployment if the build took 10 minutes that buggy code stays in prod for at least those 10 minutes while the reverted change is building.

Docker Build example: slow build

Let’s look at the docker below, this innocent-looking docker file is taken from a Node Js API. It has one major issue we will uncover as we proceed:

FROM node:14-alpine

WORKDIR /src
COPY . /src
ENV NODE_ENV=production
RUN npm install --production

EXPOSE 8080
CMD ["node", "index.js"]

RUN npm ci is another better option in place of RUN npm install --production

Let's use the regular docker build

When we try to build the above docker file with docker build using the following command

time docker build -t node-14-first-bad-cache-no-buildkit .

The time command is prefixed to the docker build command so that we know the time it takes for the docker build command to finish. Below is how long it took:

As seen above it took 57.17 seconds.

Easy speed up, use BUILDKIT

Docker build has recently added BUILDKIT from version 18.09. Docker basically says it is an overhaul of the build process. As mentioned in this post it is faster, efficient, and concurrent. You can read more about its goodness in this article on docker.com. For now, let’s see it in action:

time DOCKER_BUILDKIT=1 docker build -t node-14-second-bad-cache-with-buildkit .

As you can see the build time is less than half of the previous build without buildkit.

This build only took 27.32 seconds compared to the above build which took 57.14 seconds.

Docker Build example: fast build

Ok, there is a major issue in our previous docker file. The docker cache is busted on each change be it our custom code or any other npm modules being added. Read more about docker build cache in this post too.

Faster docker build with proper caching

Our code changes almost every time but the npm modules we pull in change infrequently. So we can safely cache the npm modules as below:

FROM node:14-alpine
WORKDIR /src
COPY package.json package-lock.json /src/

ENV NODE_ENV=production
RUN npm install --production

COPY . /src
EXPOSE 8080

CMD ["node", "index.js"]

You can have a look at the diff between these two docker files here. The main change is that we copy the package.json and package-lock.json file first then run npm install. Only after that, the custom code is copied to /src. So if you don't add a new npm library the cache will hold up.

It took 34 seconds to build for the first time as below with the following command:

time DOCKER_BUILDKIT=1 docker build -t node-14-third-good-cache-with-buildkit .

Is docker build fast after code change?

For this docker build example, I added a line of comment in the index.js file of the Node JS API application. Now let’s see how long it takes and if it caches the node_modules used in the npm install command.

time DOCKER_BUILDKIT=1 docker build -t node-14-fourth-good-cache-file-change-with-buildkit .

The build took only 6.01 seconds, thanks to great cache usage by docker and the use of buildkit.

Even though the code changed but the NPM modules were cached making the build complete in mere 6 seconds. The same principles apply for exploiting docker build cache. It can be applied to PHP with composer.json and composer.lock file or any other language. Always think of the previous command run and how can it be cached better.

All four images were around 233 MB, one took ~60 seconds and the last one took 6 seconds. That is like 10x faster.

Conclusion

If you are building docker images don’t forget to use BUILDKIT, it is super-efficient. On top of BUILDKIT always analyze how to exploit docker build cache to your advantage of faster docker builds.

I hope this small docker build example has helped you. Things like having smaller docker images like using alpine base Image can also help a bit in speeding up your docker build.

How to start a tech startup for exactly $0 with Google Cloud Platform and other services

Geshan Manandhar — Fri, 25 Dec 2020 23:39:46 +0000

Learn how to start a tech startup with no money. You might think I am joking but I am not. In this no-fuss technical guide, I will walk you through the process to start a tech startup. This guide will not dive deep into the technical details. If you know technical things like how to set up a DNS, dockerize your applications, etc you can glue together a tech startup for no cost at all.

The $300 free credit for 12 months is key to getting your startup off the ground for around 9-10 months. Let’s get started:

How to start a tech startup
Considerations
Steps
1. Domain Name
2. DNS, security and caching
3. The brochure site
4. Cement them together
5. Time for Google Cloud Platform (GCP)
6. Dockerize your application
7. Database
8. Other GCP services
9. Other free services
Conclusion

How to start a tech startup

You have an idea for a tech startup in your mind for some time. You have fiddled around and maybe have a proof of concept or even a minimum viable product (MVP) ready. Now it is time to get your startup out to the world. Or following the lean startup methodology you want to see if there is a demand for your product. You also want to know if people are willing to pay for this type of service.

For both the above cases, you will need a brand name and a working website or web application. This guide will help you get up and running at $0 stitching together the GCP products and other free services.

Considerations

You will need to register for a new GCP account and bag your $300 credit that should be used by 12 months.
You will have to register for other services as you follow along with the guide.
The aim is to run your staging and production applications for ~ $1 a day, so the $300 will last for just over 9.5 months.
After that you will need to find a way to generate money or get more Google Cloud Platform (GCP) credits like from Google for startups or some other way.
For this guide we will consider having a brochure JAMstack website hosted on Netlify
The web application will be hosted on Google Cloud Run with the MYSQL database on Google Cloud SQL.

It is time to dive deeper into the steps:

Steps

Before we go deeper into the steps, these steps are going to be technical but not deeply technical. These steps are going to be more like pointers than a step-by-step process with screenshots in hyper details. If you have found the idea till now to be interesting carry on:

Domain Name

Your brand cannot exist without a domain name. This is where you might want to spend some money if you are more serious about your tech startup. Still, if you want to go the $0 path here is your option. You can register your domain at FreeNom for $0. You can get domains ending with .ml, .tk, .ga, and.gq.

The good things are you pay $0 and there is no competition as getting a .com or .net. The flip side is you don’t get a “standard” domain as such. The choice is yours, if you are on a strict budget with $0 I would suggest to get a .ml domain and brand your startup as a “Machine Learning” idea, it has the potential to sell too :).

DNS, security and caching

The next step after you register the domain is to get a free Cloudflare account. You can then use the DNS provided in your Cloud flare account at FreeNom to direct the website/web application traffic to where you want. In our case, we will like to redirect the brochure JAMStack to Netlify. The web application traffic will go to our application hosted on Google Cloud Run.

For now, be able to log in and “Add a site”. Then Google more about how to use cloud flare to linkup the traffic to respective services. On top of DNS management, you will also get things like caching and security for free with Cloud Flare. Again, I will leave you with more Googling to get all these configured correctly. You will also get free HTTPS with Cloudflare no need to wrangle with certificates and settings.

The brochure site

Now as you have your basics setup, it is time to get your brochure JAMStack website up and running. You can follow this step-by-step tutorial to get you JAMStack website running without a line of code. The main glue here is Stackbit. With the recent changes, you could directly go to Stackbit, select a theme, and deploy your website on Netlify. You will also get an easy to use CMS with a WYSIWYG type editor that works not only for text but for images, layout, and configuration of your website.

If your startup is a SAAS product I would recommend using the Azimuth theme. It suits very well for SAAS products and with the click of a button, you can even change the color scheme.

Cement them together

After you have your brochure website running on Netlify with Stackbit. Configure the DNS correctly on Freenom so that the website traffic hits the Netlify website with Cloudflare security enabled. You can scratch the surface to connect Freenom and Cloudflare with this guide.

If you just want to test your idea then you can stop here. With the brochure website, you can start gathering email addresses using Nelify forms. Netlify forms free version has limits so please be aware of it, refer to the Netlify Pricing page. After that, you can figure out if people will pay and should I actually write for the tech startup.

Take note that, for all the above services you have not even spent a penny. So you can run your experiment as long as you want. If you have already some code and some form of your tech startup working then proceed further to host it with GCP with serverless containers on Google Cloud Run.

Time for Google Cloud Platform (GCP)

Now is the time to get your Google Cloud Platform (GCP) account. This is where you will need to search for your credit card. Even though you will not be charged but Google will take in your credit card details.

We will be using serverless containers for your application deployed on Google Cloud Run. We opt for serverless containers because of the cost and other reasons. The other reasons mainly include no need to learn a new framework/paradigm and you can run any application as long as it can be containerized. You will also get free HTTPs and custom domain mapping to point the application back to the FreeNom domain.

With Cloud run you get 2 million requests a month free. After that, it is 0.40 $ per million requests. Well, the math is not that simple but it is advertized like that for sure. Google cloud run is a lot easier to set up than a full-blown Kubernetes cluster with similar benefits. Know more about it in this talk.

Dockerize your application

To deploy your application, you will need to dockerize it. In this step-by-step guide you will see how to deploy a Laravel application to Google Cloud Run Fully managed. Following similar steps, you can deploy any PHP application to Google Cloud Run. Just bear in mind that you choose a lower-cost region like us-central-1 so that you don’t burn the free credit fast.

With my experience, currently, for an application with both staging and production workloads, it is costing us 10-20 cents a day. This is also going from our free credit.

Database

Just having some code doesn’t make the application work, the heard of most applications is the data. You can store your data on Google Cloud SQL over MySQL or Postgres SQL. This will be your main money hogger. With the current MySQL [pricing], using two db-f1-micro it is costing us 88 cents a day. 44 cents per day for the staging database and the same for the production one. There is an automatic backup setup for the production one.

To save cost, if you can live with a maximum of 5 connections at a time you can choose to use remote mysql. If you use remote MySQL for staging your daily cost for the database can come down to 44 cents a day.

One important thing is to keep your Cloud SQL instance behind a firewall. Follow the official instructions to connect Cloud Run fully managed services with the Cloud SQL database.

If you use other forms of data storage the costs will vary depending on what you choose.

Other GCP services

In addition to Cloud Run and Cloud SQL, you should use other services to make your life easier. You can use Cloud Build to build your containers. It can be linked with Github and you can start building your containers on each push or with some conditions. Cloud build has 120 mins per day free. It should be more than enough for a budding startup like yours.

With Cloud build the container images will be pushed to Container Registry. Google Container registry uses Cloud Storage to store the images. Set lifecycle configuration on the cloud storage bucket so that files older than X days can be auto-deleted before it mounts up the cost.

You can also make use of the free VM - F1 micro instance to do small tasks. We are using it to run a bot built with Hubot. It uses the gcloud command to do things like deploy cloud run applications and display logs over Slack. So needless to say we are using GCP logging and you should use it too.

Of course, GCP has a plethora of services but don’t get into the analysis paralysis mode and get nothing done. We used the above services and our product is running. Explore optimally and use the services needed for your product.

Other free services

The following are other free services that will help you build your tech startup:

Github - Github has a free team plan so don’t forget to exploit it. With 2000 minutes for Github Actions you can use it as your CI/CD pipeline too.
Trello / Asana - Depending on your project/product management flavor you can use Trello or Asana free version to manage your product development.
CodeClimate - To keep a tab on the health of your code you can use CodeClimate free plan with 4 seats. This will be a boon for your tech startup’s code quality.
ForwardEmail - Great you have your domain but using a Gmail email won’t work best for your brand identity. Get your info@yourdomain.ml for free with ForwardEmail free plan.
Free CDN - If you are serving images or static assets on your website or web application you can use Statically to get the CDN optimizations and benefits. Try it, it is free and unlimited too.

If you know of any other services that can help a budding tech startup please do add it to the comments section.

Conclusion

You don’t need money to get your tech startup off the ground. If you have the right idea and utilize your tech skills correctly you can get your tech startup started for $0.

I hope you will not let your tech startup idea die and get something running for no cost with GCP and other services.

3 must-have qualities of a great software engineer

Geshan Manandhar — Tue, 22 Dec 2020 04:50:14 +0000

There are many qualities of a great software engineer that will make you stand out. Among them, these 3 are crucial for you to become a great software engineer and leave the competition behind. Surely technical qualities matter a lot to be counted as a superior software engineer. On the other hand, this post is geared towards the non-technical aspects of software engineering.

Software engineer qualities to work on

With no doubt, a great software engineer writes working, testable and maintainable code. Numerous technical skills will be required to differentiate yourself as a great software engineer. Writing clean code, debugging bugs, and following standard are some of these technical skills.

Some opine positive attitude and communication skills as must-have qualities of a great software engineer. Another post points to curiosity and a love of learning to be essential to be an efficient software engineer. You can find a great software engineer in Eastern Europe too if you are looking for one.

For this post, we are more concerned than just the hard skills needed to prove yourself as a valuable software engineer. Let’s proceed to the software engineer qualities you need to work on to set you up for success:

Good business acumen

Simply put, business acumen is an understanding of how a company makes money and meets its goals and objectives. I have written in a previous talk too:

Solutions and value to the business is much more important than the latest language/framework.

As a software engineer, you must not worship a particular language or framework. Similarly, you should not apply the latest trends and fads to your day to day work just because it is trendy. Always think of what value does it add to the business. Also, analyze, how does it help make/save money for the business.

Think of the trade-offs and the cost to benefit ratio too. For instance, once we became aware of a bug. It was affecting ~0.25% of the customers.

The cost of fixing the issue at that time was far more than not fixing it. We did eventually fix the small bug but it was only in the next sprint.

This is an example of good business acumen as the business was not losing any money. Yes, a fraction of the customers were facing a minor inconvenience but that was also fixed in about a week.

Product managers/owners must have great business acumen. It is also important for software engineers to have good business acumen. I have never seen a software engineer who was promoted and had bad business acumen. Maybe their technical skill was a bit rusty but it was easily covered by good business acumen.

If you work as a software engineer for an e-commerce company, you must know what the conversion rate is. Similarly, it is better to know things like Customer acquisition cost (CAC) and Customer Lifetime Value(CLTV). In case you are working for a SAAS business, you much understand what churn rate is all about to sharpen your business acumen. You can learn more about how investors look at numbers in a fun way by watching Shark Tank. It sums up to having a great sense of how the business works and accumulate valuable domain knowledge over time are super crucial qualities of a great software engineer.

Highly responsible

Great software engineers are highly responsible. If they take up a task, they make sure that it is working well on production. They will even go above and beyond to establish more clarity around the task by building reports for better visibility.

I have never met a great software engineer who will throw his/her work over the wall. By that, I mean saying things like “It is done from my end now it is QA’s responsibility” or “I have written the code, it is approved. Now it is DevOps’ headache to get it to production”.

A quality of a great software engineer is that they follow up the task on hand very well. Even after the task is done they will come back to it after days or even weeks to check that is it working fine. They will evaluate numbers and take pride in the great work they have shipped.

For an efficient and highly responsible software engineer the Quality Assurance (QA) team is more like a gate to pass by not a group of testers.

The engineer would have already thoroughly tested their work in both automated and manual ways. Finding obvious bugs in the work done by an amazing engineer would be difficult. As they would be already highly responsible to think about that edge case or that probable use-case and tackled it in the code. They might have gone ahead to write tests for such scenarios so that it doesn’t come back and bite the team in the future.

Following true DevOps and SRE philosophy, a great software engineer will own their work and be accountable for it.

If things break in something they have developed and deployed, they will be the first to respond.

The response can be as quick as a rollback or even write a quick fix to patch the issue temporarily. They will come back to the issue and release a permanent solution to the issues introduced by the code they have added. Basically, great software engineers truly live by the philosophy of you build it you run it. They also make great on-call engineers.

Ace team players

Being an ace team player is another crucial quality of a great software engineer. Engineering resilient software is a team sport and efficient software engineers know this fact very well. A great software engineer is an excellent mentor and a coachable mentee too.

A great software engineer will always look out for the whole team. They will help engineers junior to him/her and not hesitate to learn from everyone. A valuable software engineer will logically help another teammate. This does not mean that software engineer will always leave their talk on hand, context switch, and help another engineer. This means given the task on hand is at a parkable state they will park it, then go and help the junior team member get unblocked.

Great software engineers have this innate ability to translate technical jargon and requirements into simple layman language.

This in turn helps all team members understand what is being done. For instance, a non-technical product owner can easily explain to the executive why this feature will take more time. A valuable software engineer would go the extra mile to explain the need to refactor and then only do the main task. This will make it faster in the long run and make the feature more reliable too.

A great software engineer is an impact multiplier. Let’s say if s/he is somewhat free and they see that the QA person has a lot to do in this sprint. They will not think twice about helping the QA person and help the QA person.

A great software engineer knows and values the team output. S/he contributes to the team’s flow and outcome to be better not only their own.

Imagine this, the sprint is near to an end but nothing is deployed to production even though pull requests have been approved. A software engineer who is a valuable team player would assist in things that would help code reach production. They will be able to do it without overstepping on anyone’s responsibilities. These software engineer qualities are found in great software engineers.

Conclusion

All of the above will only foster if the Software engineering culture of the company is on the right path.

As Dr. Willam Tate mentioned - “Rarely do managers focus on the quality of the fish tank & what surrounds the fish: they mostly notice individual fish and become fixated on them. But if the water is toxic, the fish suffer. If there is no movement in the water, it will be deprived of life-giving oxygen”

To get the best qualities applied even out of great software engineers the whole software engineering culture needs to support it.

Create small pull requests by using enabler code last

Geshan Manandhar — Tue, 22 Dec 2020 04:33:02 +0000

There are multiple ways to create small pull requests for a single task. This post discusses an easy way to do it which has no impact on production code. The steps are laid out in a way that small pull requests will be reviewed faster with no effect on production too. Let’s see how that happens with enable code last:

Importance of small pull requests

Without doubts, small pull requests are efficient and effective. Big pull requests with >200 lines of code have multiple disadvantages. As per a study, longer it takes for the reviewer to find a big enough slot of time to address it.

Similarly, larger pull requests have less chance it will pass code review on the first attempt and therefore require multiple reviews.

On the other hand, as Donald G. Reinertsen mentions in his book The principles of product development flow smaller batch size reduces cycle time and accelerates feedback.

Smaller batch sizes also reduce risk and overhead. Thereby, small pull requests also have the same effect.

As per Github's Octoverse 2020 Productivity report on page 5, the first key finding is:

Small pull requests drive innovation and productivity

It further adds, "Teams that focus on small pull requests and closer collaboration have better reviews and faster feedback. Throughout the year, developers stepped up their work by keeping pull requests at the same size or smaller and merged pull requests up to seven and a half hours faster. This gives developers more time to do the things they love". Yet another data-backed reason to have smaller pull requests.

In my personal experience, I have opened pull requests with 50 files changed and 2 files changed. Of course, the one with only 2 files changed and ~100 lines changed got merged a lot faster.

Role of enabler code in small pull requests

So, great to know smaller pull requests get merged faster but the most important question is how to make the batch smaller. Making a pull request smaller makes it a lot easier to review. This results in getting approval faster, which means the cycle time is reduced.

Without doubt, large batch sizes reduce efficiency and the same definitely applies for large pull requests.

I have mentioned this in my previous post about how to get your pull request reviewed faster too, this time below is a practical example.

Some considerations

The example below applies only to web applications (probably most of the software the readers of this blog write is web development in some way or form)
As soon as we talk about a website or web application, the cement that glues together the path/URI with code (generally a controller) becomes a pivotal part of this approach
For this solution to be really useful, utilizing a form of automated testing will be necessary. Unit tests would be an optimal choice in this case.
Given example is of a backend API. Still, the same concepts can be easily used for a frontend application or any other web application.

Example of small pull requests

We will be using an example application that gives out currency conversion rates. The application is open source and already running. For this example, we are adding a new feature to list all the rates for a given currency with pagination.

As we want to have small pull requests we will divide the task into two pull requests.

If you have a big system with code spread across over multiple files it can be broken down into more than two smaller pull requests.

It will also be advisable to have new files/classes into different small pull requests. As these files will have their own tests it will be a lot easier to review and get merged.

The main trick here is in the last pull request with enabler/glue code. Until the last pull request with the enabler code that stitches all the code added prior to it with a route is not deployed there is no effect.

Let’s see how to do this.

First small pull request

The first small pull requests will have the functionality to list all the rates for a given currency. It will also have pagination implemented. We will verify that it works with unit tests.

A better way to do this first part would be to write the functionality without pagination. It will have tests for this feature without pagination. The next step could have been to add the pagination part and the related tests or modify existing tests to accommodate the pagination changes in another small pull request.

This is a very small application and it is not written in a SOLID way. So, our change to show all exchange rates for a given currency is incorporated in this small pull request. The main change is give below:

async function getByToCurrency(currentPage, currency) {
  const offset = (currentPage - 1) * config.itemsPerPage;

  let currencyExchangeRates = await db.query(
    `SELECT from_currency, to_currency, rate, on_date FROM exchange_rates where to_currency = ? LIMIT ?,?`,
    [currency, offset, config.itemsPerPage]
  );

  if (currencyExchangeRates.length) {
    return currencyExchangeRates;
  }

  return [];
}

As you can see, the change is merely 14 lines of code which is very straightforward. Then, there are 80 lines of unit test code to verify that it works correctly.

Notice here that this code can be reviewed and merged but it will not affect anything in production.

It is possible because the code is not reachable as of now. There is no route or controller action to reach this new piece of code.

Small pull request with enabler code

Once the above small pull request is merged and deployed you can start work on the enabler cement code. This cement code wires up the above code to a user action like viewing the rates. It will be possible with a route that will invoke the above getByToCurrency method. You can see how we exposed the code as a new route in this small pull request. You can see the main code below:

app.get('/api/rates/:currency', async (req, res) => {
  res.json(await exchangeRates.getByToCurrency(req.query.page || 1, req.params.currency));
});

These 3 lines of enabler code act as the cement between the user hitting the URL and linking the new route to our existing code from the previous small pull request.

Following these simple steps, you can also learn the art of small pull requests.

Not only will small pull requests be efficient and improve your team’s cycle time. It will also have no impact on production unless the final piece of code is deployed and released.

To lessen the impact of the code and control the blast radius you can surely use feature flags. You can have a look at an ultra-simple example too.

Conclusion

Software development does not need to be difficult. If you break down the tasks into bite-size pieces, manage expectations well, and plan your tasks accordingly it will be much easier.

Don’t underestimate the power of a smooth flow enabled by amazing cycle time. Desirable cycle time can be achieved by using small batch sizes. Small pull requests will definitely make your batch size small.

The next time you open a pull request ask yourself if it is a small pull request or can it be made logically smaller.

What is Agile software development? Explain to me like I’m five

Geshan Manandhar — Sun, 08 Nov 2020 23:51:07 +0000

Agile software development means different things to different people. In this post, we will define what agile software development is with a simple analogy that a five-year-old can understand. The analogy we will use is as straightforward as serving an apple to 4 kids. Yes, it will be that easy, let’s get started:

What is agile software development?

Atlassian, in their guide, defines agile as:

Agile is an iterative approach to project management and software development that helps teams deliver value to their customers faster and with fewer headaches. Instead of betting everything on a “big bang” launch, an agile team delivers work in small, but consumable, increments.

Wikipedia has an article on Agile software development, that says:

In software development, agile practices approach discovering requirements and developing solutions through the collaborative effort of self-organizing and cross-functional teams and their customer(s)/end user(s). It advocates adaptive planning, evolutionary development, early delivery, and continual improvement, and it encourages flexible responses to change.

Until now it is not understood what agile software development actually means. Have a go at it:

Agile software development means developing software in such a way that the end-user feels value is being delivered frequently in the form of working software. This surely includes having the end-users get involved in the agile software development process.

The software engineering team does not take months to write up requirements, then again months to design/architect the software. After that, the team writes the code and delivers the product to the customer. The end-user is part of the process. After the end-user gives some requirements they are able to see part of the working software rather than waiting for the whole thing to finish.

Rather than that, the team understands the most important part, gets requirements for that section, designs/architects it, builds it, and delivers usable software in weeks, not months or years. That is agile software development, in a nutshell. Don’t be agile just on paper, implement it practically.

Agile Manifesto

Around 20 years back in 2001, 17 men met in Utah and wrote up this amazing manifesto as guiding principles for agile software development. There is a long backstory to the manifesto, if you are interested in it, please read it. Below are the 4 main values:

** Individuals and interactions over processes and tools

Working software over comprehensive documentation

Customer collaboration over contract negotiation

Responding to change over following a plan*

You can always argue how relevant these points are if you work in an agency setting where you do customer projects, not company products. At an agency, you need to come up with a contract with the cost of the project. That makes most of the above values hard to meet. Anyhow, these are the guiding principles for agile software development.

In addition to the above values, there are 12 principles of agile software. The first one being:

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Then there are other principles discussing changes, delivery of software, collaboration, etc. So until now it still seems jumbled, right? This is surely not something a five-year-old can comprehend in any way. Follow on to read the over-simplified analogy.

Agile software development analogy

Now let us move away a bit from agile software development and all the jargon that it entails. Allow me to introduce a very simple problem (a bit dramatized) which is a great analogy and example. The example here is to serve an apple to four hungry children.

Imagine this, you have four hungry children and one apple. You need to serve that apple to these 4 children as a snack while the main course is on its way to be delivered. All four children only eat peeled, cut, and diced apple. How can you go about this issue?

Waterfall approach to serving an apple

If you follow a waterfall software development, you would do the following steps:

Wash the apple you have got
After that, cut it into four similar-sized pieces (halve the apple then halve both the halves)
Peel all the four pieces of apple one by one
Cut the seed out
Dice all the four pieces of apple
Put the dices in four plates
Then you serve it to the four children

Now, notice some of the things that happened in this process:

None of the children could satisfy their hunger until step 6 was done
There was no way to give feedback about the size of the apple’s dices, you had to eat what you got
The children got served all the apple dices of their portion in one go. This may be a good or a bad thing depending on how you understand it.
Surely there was no feedback loop in this process, the children felt a bit left out.

All in all, the process looked normal. Unfortunately, software development is not as easy as serving an apple to four hungry children. So let’s have a look at how agile software development would attempt to solve the same problem.

The agile software development approach to serving an apple

If you had followed one of the agile software development methods, the process would look more like below:

Wash the apple
After that, cut one quarter out of the apple
Peel that cut out one-fourth part
Cut out the seed from the one-fourth part
Dice it into eight pieces
Put 2 pieces each in 4 plates
Serve the two pieces each to the four hungry children
Get feedback, go to step 2, and repeat until the apple is finished :)

Again, let us take note of the things that are better this time:

The children got served a lot faster even though it was a smaller portion of apple
They had the opportunity to say if the dice was big or small, so in the next iteration, the size could be customised following the child’s wish. This is a simple example of early user feedback.
Every iteration could be better in terms of the size of the dice if the apple was sour the children would know it faster. These are the advantages of early feedback.
Another important aspect, if the children are not hungry anymore after 4 dices, you can stop and have not wasted the whole apple.
This approach also enables a test and learn phase. You could easily give one child smaller apple dice if the child was having any issue chewing bigger pieces of apple.

On the whole, the agile software development method was more collaborative and provided more satisfaction to the four children.

It provided more satisfaction not only by serving the apple dice faster but also by letting the children have a say in the next round of dice they would receive. That is the beauty of agile software development.

Drawing parallels between the analogy and agile software development

You can easily draw parallels between both the processes of agile software development and serving the apple. Requirements analysis can be mapped to washing the apple.

Similarly, software design and architecture can be linked to cutting the apple. If you don’t cut the apple correctly, it will be difficult to get well-shaped dices and so forth. The same goes for software development.

Even serving the apple can be associated with deploying the software. In agile you will deploy and release software early providing time and scope for feedback.

Conclusion

This is an over-simplified analogy between software development and serving an apple to hungry kids. Still, I think you can easily identify the advantages of doing agile software development with this example.

Of course, software development work is much more complicated than serving an apple to four children.

Always think of the smallest possible value a software engineering team can deliver to the customer. This will create a win-win situation for everyone involved.

And yes, even a five-year-old can easily understand that agile software development is better as it will satisfy the hunger faster.

DEV Community: Geshan Manandhar

Set Up Tracing for a Node.js Application on AppSignal

Why You Need Tracing and an APM Tool: A Case Study

Tracing and APM for a Node.js Application

Add AppSignal to the Node.js Project

Run the Node.js App with AppSignal Reporting

Sending Requests to the Node.js App

Analyzing Stats and Graphs

Checking Memory Usage

Visualizing Response Times

Know Your Slow Queries

Tracking Errors

Other Useful Features in AppSignal

Wrapping Up

How to Handle Async Code in JavaScript

Async Programming: A Quick Intro

Callbacks with Async JavaScript

Callback Hell in JavaScript

Async JavaScript with Promises

Async with Await for Promises

Make Your JavaScript Async Code More Efficient

Wrap Up

How to outclass your competition in take-home coding challenges

What is being assessed? #

Steps to outclass your competition in take-home coding challenges #

Limit the scope and time investment #

Plan and do step-by-step pull requests #

Write automated tests #

Dockerize it #

Write a killer readme #

Conclusion #

3 Software Engineering podcasts you must subscribe to now

Command Line Heroes (from Redhat) by Saron Yitbarek

Developing Up By Mike Miles

The SAAS Podcast by Umer Khan

Conclusion

Node.js Postgresql tutorial: Build a simple REST API with Express step-by-step

Table of contents

Prerequisites

Steps for Node.js PostgreSQL tutorial

Setup Express with express generator

Delete the public folder

Delete unnecessary existing routes then create a new route for quotes

Change index route to give out JSON

Setup PostgreSQL with quote table

Link Node.js with Postgres

Show Quotes - GET API with pagination

Save a new quote - POST API for Node.js PostgreSQL tutorial

Adding validation for creating quotes POST API

TLDR; quick rundown

Conclusion

How to make PHPUnit Code Coverage 2+ times faster with Pcov compared to Xdebug

Save time on CI builds

PHP code coverage with Pcov, not Xdebug

Considerations for code coverage with PCov

Procedure for faster PHPUnit code coverage

Fork Laravel/framework repo and run tests only for PHP 7.4

Run PHPUnit code coverage with XDebug

PHPUnit Code Coverage with Pcov is 2x faster

Quick comparison of Code coverage

Conclusion

Working as a software engineer on internal products Vs a customer-facing one in E-commerce companies

Why e-commerce examples?

Assumptions

What is an internal product?

What is a customer-facing application?

The differences in working on internal applications vs customer-facing ones

Fewer performance and scalability considerations for internal applications

Customer-facing applications are more security sensitive

Minimal use of feature flags, A/B testing, and gradual rollouts on internal applications

Customer-facing applications get more UX love

Summary

Conclusion

Docker build example: how to go from slow to fast docker builds

Information before jumping in

Why faster docker build

Docker Build example: slow build

Let's use the regular docker build

Easy speed up, use BUILDKIT

Docker Build example: fast build