DEV Community: Alfonso

Improve Your Debugging Approach for Better Software Applications (& Sounder Sleep 😴)

Alfonso — Mon, 06 Feb 2023 15:37:18 +0000

A page from the Harvard Mark II electromechanical computer's log, featuring a dead moth that was removed from the device with the note "first actual case of [a] bug being found", on September 9, 1947.

Introduction

In the process of developing applications, bugs will inevitably be introduced (“I code, therefore I create bugs” ~ Descartes, probably). Bugs can be introduced for a variety of reasons, such as logical errors, misunderstanding of requirements, lack of tests, tight deadlines, or something as simple as having an off day as a fallible human. However, knowing that bugs are inevitable, we can arm ourselves with tools to quickly identify and address software bugs before they are released in the wild.

Many people assume that they already know everything that there is to know about finding and fixing bugs, or that debugging can be an afterthought since they believe that feature development should take priority. The reality is that it is all too common to approach debugging with a haphazard, ineffective approach. Haphazard approaches can lead to frustration and wasted time, and can actually prevent you from working on new features for your users.

What a haphazard debugging approach might result in

In this post, we will break down what debugging is as well as one approach that you can employ while you are developing software applications. Additionally, we will add more tools to your debugging toolbox that you may not be aware of. We will also review some interesting bug case studies, noting how the authors employed some debugging techniques in their approaches. This post assumes some familiarity with programming concepts.

What Is Debugging?

If we accept that creating software inevitably leads to the creation of bugs, then we must accept that some of the time taken in software development must be allocated to maintaining and debugging existing code.

I caught a bug!

Debugging is the process of locating, identifying, and fixing bugs. Even though a test might reveal the presence of a bug, it will not tell us what the exact error is or how the code needs to be fixed. Oftentimes, developers will approach debugging in a sub-optimal way: randomly opening files in the codebase in the hopes of finding out where the issue is coming from; changing lines of codes in a seemingly random manner and restarting servers in the hopes that their changes have fixed the issue; or worse yet, being paralyzed into thinking that the code should not be touched further in fear of causing other unintended consequences.

Why Is Debugging Important?

With great software comes great responsibility. It is a disservice to our users to ship products with glaring, obtrusive bugs, as these bugs can lead to unexpected results. Depending on the industry that you are operating in, software bugs can lead to financial losses, reputational damage, loss of trust from your users, or personal injury, or they can live in infamy for causing a security vulnerability.

The following famous bugs demonstrate the importance of squashing bugs before they are shipped out to users:

NASA Mars Polar Lander - the lander was destroyed because its flight software mistook vibrations caused by the deployment of the stowed legs for evidence that the vehicle had landed, and shut off the engines 40 meters from the Martian surface. The lander had traveled approximately 79.5 million miles up to that point, only to crash and fail at the end. This resulted in financial damages of $175 million and a failed mission.
Therac-25 - a computer-controlled radiation therapy machine that, due to several software bugs, incorrectly administered massive overdoses of radiation, resulting in the deaths of several patients. A quintessential case study of the potentially fatal dangers of engineers’ overconfidence.
Knight Capital Group - Knight Capital’s systems incorrectly executed trade orders due to a repurposed software flag that triggered defective code. This resulted in a loss of $440 million.
“MissingNo” Pokemon - a glitch Pokémon species present in Pokémon Red and Blue, which can be encountered by performing a particular sequence of seemingly unrelated actions. Capturing this Pokémon may corrupt the game's data.

This bug can be exploited to duplicate items and modify game behavior

A Debugging Approach

Below, I will break down my personal debugging approach. I lean on principles that I have learned in other engineering disciplines, years of observation while pairing with other developers, and utilizing resources on how to improve troubleshooting skills.

It is important to note that this post assumes that we have not already prevented and detected certain software bugs with error handling, testing, linting, static type checking, proper code formatting, and additional assistive tooling.

I like to follow the scientific method approach when I am debugging:

Make an observation
Gather information
Make a hypothesis
Test my hypothesis
Analyze if my test is or is not working
Repeat until bug is fixed
???
Profit

For step 1, making an observation, I first check to see if the buggy behavior still exists, or if it is actually a bug at all—bug reports might be incorrect or users might be reporting an issue inaccurately. At times, I will pick up a bug ticket but when I go to verify the buggy behavior, the bug has already been fixed by other work. It’s important to verify that the behavior still exists before spending time on an unnecessary fix. If the buggy behavior is still present, I verify that it also occurs in my local development environment. This helps eliminate any subtle issues that could be present only in production due to differences in environment configuration.

I find that the bulk of the work should be done in step 2: gathering information. This is where it is useful to know what tools you have at your disposal, including:

Console.log
VS Code Debugger (or the Chrome Debugger, or the debugger in the editor of your choice)
Chrome dev tools
React dev tools
Redux dev tools
Service logs, terminal logs
Your peers
Rubber ducks

As developers, we are lucky that there are a variety of tools to help us with data collection. Debuggers, logs, and additional dev tools should all be utilized to track down as much information as possible about the bug. They can really help narrow down the scope of the problem. I would implore anybody reading this post to explore these tools in more depth, as updates are constantly being made to improve the developer experience. There is an abundance of resources available regarding any dev tool that you may be using. These tools will help you throughout the entirety of your career, so learning them well is a very good investment of your time!

For step 3, making a hypothesis, it is important to eliminate any assumptions that you are making about how the program is operating. Coming up with a hypothesis on the root cause of the bug should involve clear thinking. Random guesses here will not be helpful so be sure to take a step back and use all of the information that you gathered in the previous step to make an informed decision. Really tricky bugs, especially those lacking error messages, might need to be tackled with informed trial and error.

Step 3, making a hypothesis, in brief

After coming up with your hypothesis, you can move onto step 4, testing your hypothesis: you can narrow in on the part of the codebase that you suspect contains the bug. You can then add debugger statements before the suspected areas of code to step through how variables and functions are operating, comment or modify specific sections of the code, or create unit or integration tests that account for the buggy behavior and re-run your test suite.

Along with step 2 (collecting information), I believe that step 5, analyzing whether your test is working, is next in order of importance. After you have modified sections of the codebase, you should examine your changes. A careful analysis can help you determine if this bug is isolated to this section of the codebase, or if the buggy behavior might be present in similar areas in other parts of the codebase. If the buggy behavior is still present despite your changes, you need to cycle back up to step 1 and repeat this process.

After performing these steps on a variety of bugs, you will automatically start to cycle through these steps when you encounter any new issues. This can help with bugs that are reported, but it can also help with catching bugs before they are ever committed into the codebase.

The great thing about this approach is that it can also be used while pair programming. Having multiple sets of eyes on a problem and cycling through these steps can help you find the issue more quickly. I have found that the “time to ask to pair program” threshold is different for everyone, but it is important to leverage your teammates and their knowledge. You can ask yourself who has last worked on this feature, or who knows a lot about this feature and may be able to provide more insight?

Additional Tips & Tricks

In addition to the approach that I like to use, I also like to keep the following things in mind:

Check easy and fast stuff first, even if it seems unlikely. For example, are my servers running, am I checking the correct file, does the TypeScript server need a quick restart? Did I save my changes?
Depending on the type of bug, do I have the relevant screens up on my monitors? Are Chrome Dev Tools open, are my terminals open, are any back-end logs open, are the relevant dev tools open (e.g., React dev tools, Redux dev tools)?
Is this a rendering issue (React/CSS) or a logic issue (JS/TS)? Both?
Is this a front-end issue or a back end-issue? Both?
How long has this bug been present? I often like to use git bisect to detect which commit introduced the bug.
Make sure to read/skim the entire error message. However, logs often contain lots of noise.
Error messages are clues but not gospel, and they can be misleading. For example, error messaging might be incorrect, it may not have been updated properly by a previous developer, or the error could be coming from further upstream. Error messages can be red herrings, and it is important to treat them as such.
Have other people experienced this issue? Check Stack Overflow and GitHub issues using key search terms.
Are there any special notes in any of the relevant documentation that may have been overlooked?
Use the “fold”/“unfold” features of your editor to minimize the amount of code noise that your brain must process while you are skimming through files.
Taking a walk—sometimes stepping away and coming back with a fresh set of eyes leads to new insights. In the same vein, those “eureka!” moments could come when you are away from your machine.
Make use of “rubber duck” debugging - explain the problem to an inanimate object (or a willing coworker) step-by-step. As you present the information, you might spot the error or places where your thinking could be reevaluated.
Have you tried asking ChatGPT? Although still in its nascent stages, it can serve as a quick primer for certain questions. (Disclaimer: make sure to verify its results are correct and accurate.)
Did you try turning it off and on again? In the extremely rare event that you are experiencing non-deterministic behavior, it might not be you but the universe: The Universe is Hostile to Computers.

Asking ChatGPT about CORS issues

Crash Bandicoot: A Case Study

The culprit

In this post regarding a memory card error in Crash Bandicoot 1, the author describes how they narrowed down the problem:

“About the only thing you can do when you run out of ideas debugging is divide and conquer: keep removing more and more of the errant program's code until you're left with something relatively small that still exhibits the problem. You keep carving parts away until the only stuff left is where the bug is.”

“I returned repeatedly to the test program, trying to detect some pattern to the errors that occurred when the timer was set to 1kHz. Eventually, I noticed that the errors happened when someone was playing with the PS1 controller. Since I would rarely do this myself—why would I play with the controller when testing the load/save code?—I hadn't noticed it. But one day one of the artists was waiting for me to finish testing—I'm sure I was cursing at the time—and he was nervously fiddling with the controller. It failed. "Wait, what? Hey, do that again!"”

Recap

In conclusion, if you take anything away from this post, let it be this: debugging is a skill that can be improved. Through the use of a methodical and structured approach, you can improve how quickly you detect and address bugs. Doing so will lead to a more pleasant development experience and more robust applications for your users.

Happy coding!

Why Your Emails May Be Bouncing Back & What You Can Do About It

Alfonso — Thu, 06 May 2021 13:05:06 +0000

Recently at Giant Machines, I was investigating a handful of emails from one of our client projects that were not being properly delivered to specific Internet Service Providers (ISPs). After doing some research on their email settings and configurations, I discovered that an ISP's spam scanner configuration may have been mistakenly blocking harmless emails for its customers. As a result, I had to troubleshoot and debug these emails locally to identify why they were not being delivered. For the local debugging that I did, I used an npm package called spamscanner, which I will be covering in more depth down below.

55% to 85% of all email traffic is due to spam emails! Due to this, email clients are constantly attempting to block potentially harmful spam emails from reaching their users. However, at times, perfectly valid (and non-spam) emails can be flagged incorrectly.

For this post, I will cover how ISPs use spam scanners to try to detect malicious emails. I will also show you how to set up a spam scanner locally that you can use to debug and troubleshoot your own emails. This troubleshooting can be helpful in determining why your emails may not be reaching their intended recipients! This post assumes a basic familiarity with JavaScript, using a terminal, and using a code editor.

Email Basics

The process for sending and receiving email has changed quite a bit since its inception, but its basic principles have remained in place. For the purposes of this post, all that you need to know is that web mail clients (such as Gmail, Outlook, etc.) communicate back and forth with dedicated email servers. For a closer look at how email works under the hood, refer to the following article: "How Does Email Work."

Email Authentication

Due to the abundance of email spam, several protocols have been implemented over the years to try to mitigate spam messages by performing various programmatic checks.

The three main protocols are:

SPF (Sender Policy Framework): Is the sender who they claim to be?
DKIM (DomainKeys Identified Mail): Encrypts email headers with a private key; servers then use a publicly available key to decrypt the headers and verify the message.
DMARC (Domain Message Authentication Reporting and Conformance): Built on top of SPF and DKIM; senders can set policies deciding how to handle SPF/DKIM and what to do for failing checks.

For additional information on these email authentication protocols, refer to the following article: "How Email Authentication Works."

Email Spam Scanners

To detect whether incoming emails are malicious, mail servers also use spam scanners, such as Apache's popular SpamAssasin. The internal workings of these spam scanners can be somewhat complicated (involving Naive Bayes Classifiers on trained, large datasets, for the curious), but the primary takeaway is that these classification systems typically assign a numerical point value to an incoming email to determine the validity of the message. The higher the score, the more likely that the email is spam. For reference, the ISP Optimum states the following regarding their spam filtering:

All incoming emails are evaluated against these spam rules and are assigned a "spam score". This score determines whether the message will be classified as spam. For standard filtering, the threshold is set at 5, meaning any message with a score of 5 or higher is classified as spam. Messages scoring between 5 and 10 will be delivered, but will include a spam notification in the subject of the email so that you can immediately identify and delete these messages.

Different ISPs have different policy configurations on their chosen spam scanner, but the same idea applies.

Services like Litmus provide the ability to see how various spam scanners rank your emails.

As you can see in the screenshot above, the email template that I was investigating received very low scores across the various spam scanners. So what gives? Why were these emails bouncing back, despite having a low score? We will be taking a closer look at this specific issue down below.

First Steps

Before using a spam scanner to investigate and troubleshoot your email templates, there are some quick wins for lowering your score that can be achieved by following some of the recommendations listed in this article.

So… about that local spam scanner setup?

Initial Setup

For installation instructions on the npm package spamscanner, refer to their docs.

Simple Server and Local Email Template

Email clients allow you to download your email messages (with the file extension ".eml"). With these locally saved messages, we can run spamscanner against them to further inspect their contents.

Assuming that you have installed spamscanner and have Node.js locally setup, you may use the following bare-bones script for running the scanner against a locally saved email message:

// in a file called index.js
const fs = require("fs");
const path = require("path");
// Make sure to install spamscanner in your package.json
const SpamScanner = require("spamscanner");

const scanEmail = async () => {
  // For a list of all options & their defaults, see:
  // https://www.npmjs.com/package/spamscanner#api
  const scanner = new SpamScanner({ debug: true });

  // Swap out the "Your_locally_saved_message_here.eml" file with the actual filename in the directory
  // containing this script
  const source = fs.readFileSync(
    path.join(__dirname, "Your_locally_saved_message_here.eml")
  );

  try {
    const scanResult = await scanner.scan(source);

    // For a list of properties available for inspection, see:
    // https://www.npmjs.com/package/spamscanner#scannerscansource
    console.log("Scan results, scanResult.mail:", scanResult.mail);
  } catch (err) {
    console.error("Error in scanEmail:\n", err);
  }
};

scanEmail();

// To run this script, run `node index.js` in your terminal where this script resides.

Note that you can also run the scanner on Strings or Buffers as long as they are a complete SMTP message (i.e., they include headers and the full email contents).

The results of running this script will come back in the following shape:

interface ScanResult {
  is_spam: boolean;
  message: string;
  results: {
    classification: Object;
    phishing: Array;
    executables: Array;
    arbitrary: Array;
  };
  links: Array;
  tokens: Array;
  mail: Object;
}

For a detailed description of these fields, refer to the docs. Typically, the result in the is_spam field should be enough to give you confidence that your email will not be marked as spam. Note that spamscanner does not assign a numerical value but instead opts to return a boolean.

However, different ISPs use different spam scanners, and it may be necessary to investigate your email messages further. To do so, make sure that the "debug" flag is set to true, as per the code sample above. You can then inspect the contents of scanResult.mail, which is an object containing more detailed debugging information regarding the email contents (shown below).

This ".mail" object returns the following shape:

interface ParsedMail {
  attachments: Attachment[];
  bcc?: AddressObject | AddressObject[];
  cc?: AddressObject | AddressObject[];
  date?: Date;
  from?: AddressObject;
  headerLines: HeaderLines;
  headers: Headers;
  html: string | false;
  inReplyTo?: string;
  messageId?: string;
  priority?: "normal" | "low" | "high";
  references?: string[] | string;
  replyTo?: AddressObject;
  subject?: string;
  text?: string;
  textAsHtml?: string;
  to?: AddressObject | AddressObject[];
}

It can be used to get more specific information about the email.

A sample screenshot of the "headers" field that is a part of the ".mail" object is shown below.

In the emails that I was investigating, the spam scanner classifier was marking the email messages as "not spam" but Optimum was appending the following X-Optimum-spam: yes header to the messages as they were incoming:

Some of the headers present in the email message file. Note the Optimum spam header.

This was causing these messages to not only be marked as spam but they were also being blocked/bounced entirely!

When all else fails, try manual debugging.

If your messages are still being blocked despite a low spam scanner score (or is_spam is false if using spamscanner), you may have to take a more manual approach. To do so, I gradually removed parts of the email and re-sent the trimmed-down emails to the ISP that was blocking us. I was eventually able to trace the problem down to this line:

<a href="mailto:example@example.com">Contact customer support</a>

Specifically the mailto: present in the template caused Optimum's email configuration to flag the email as spam and reject the message outright, despite mailto tags not causing our messages to be flagged as spam by other ISPs.

Additionally, other emails were bouncing back due to the following (modified) copy:

If this email change request wasn't authorized by you, please click the link below to cancel. If you have any questions, you can contact support via example@example.com or by calling +1 555-555-5555.

Specifically, the +1 present in the template caused Optimum's spam scanner configuration to flag the email as spam and reject the message outright despite being valid and despite not being flagged by other ISPs or SpamAssasin.

Due to Optimum's unique SpamAssassin configuration, we were seeing issues for our customers who had an Optimum domain email and attempted to receive emails with "mailto:" or "+1" present. It is not clear why Optimum chooses to block these emails when other ISPs do not, but it could be the case that their configuration is particularly sensitive and errs on the side of caution in attempting to mitigate potential security risks.

The issues that may be affecting your emails may differ but the techniques used here can help you narrow down why your emails may be bouncing back!

TL;DR

Email servers accept, forward, and deliver messages.
If an email has not been properly authenticated, the email server must return a failure "bounce" message.
Spam scanners typically assign a point ranking to emails to classify them as spam or not spam. Hot dog/not hot dog anyone? 🌭
You can use the npm package spamscanner locally on your email templates to check whether they are being classified as spam.
When all else fails, you may have to try a more manual debugging approach to debug ISP-specific edge cases.

Additional Resources

Have any questions? Comment down below and happy coding!