DEV Community: Clarice Bouwer

Owning the Chaos: A Simple Guide to Tackling Obscure Errors

Clarice Bouwer — Thu, 12 Dec 2024 18:55:00 +0000

AUTHOR'S NOTE:
When something is obscure it means that it is not clear or is difficult to understand.
This implies that something is vague, hidden, or lacking in detail.
Cryptic indicates that something is mysterious or puzzling.
Some errors are both obscure and cryptic so feel free to use the terms interchangeably in this articles.

Ping! A slack message. You wake up. Oh look, it's #errors. "What's broken now?" you think to yourself. You go to the channel and see an automated message rendered on the screen. It reads something like Error: Something disastrous happened. It's obscure and it makes you pull a perplexed face. You scroll up to see more errors from the night before. This forms a deepening sickening pit in your stomach. Some errors make sense. Others are just as baffling as the original.

Adding to the challenge, you're juggling the complexity of multiple environments like development, feature spin-offs, staging and production. So, the errors that come in are from numerous contexts in the domain, applications, services and said environments.
This impacts the decision on which errors to prioritize and respond to first.

What a way to wake up—and all before breakfast 😞

With no immediate response from others, you need to decide what to do. It's still early. Do you roll over and sleep some more? Wait for someone with more experience? Rely on the expert in that part of the domain? Just jump and figure it out?

Well, it depends actually.

The severity of the errors. If they are blocking users, you need to act fast. If they are not, you have a bit more time.
Familiarity with the domain, applications, services and infrastructure. You may need to ask a few questions.
Experience with the tools and technologies.
Confidence in your ability to resolve errors.
Availability and bandwidth. If you are not mentally ready, spread too thin, burned out or have other commitments, you may need to ask for help.

This is an extreme but common scenario where there are floods of errors coming in from various sources. Sometimes errors storm in like a tsunami or trickle in like a leaky tap. Either way, they need to be addressed.

In moments like these, someone needs to seriously step up and take ownership, ensuring these errors don’t fall through the cracks. Good team habits are critical to ensure the code is written in a maintainable and supportable way, that errors are resolved in a timely manner and that team member burn-out is avoided.

This dreaded scenario is no stranger to me. I've seen some pretty weird things pop up. Therefore, this topic is hot on my radar because I care deeply about the quality and integrity of systems. To me, the answer here is that you should care. Even if you don't know how to solve it.

So, how do you tackle these challenges head-on? Let’s break it down.

Triage

Understanding the domain, the applications & services and parts of the infrastructure is important. If you are a new hire or new to that domain, chances are that you'd know less that the seasoned engineers.
As you may be ramping up, don't hesitate to speak up and ask questions. You are not expected to know everything because, in reality, no-one does. Even seasoned engineers don't know everything—experience doesn’t make you omniscient.

Read through the errors in the channel carefully.
Identify the critical environments (example production vs staging), the applications, services & jobs that are affected and how often the errors are occurring.
Triage by spotting common or related errors. Can you identify patterns, are errors originating from the same source? Is there a trend in timing or frequency? Which errors are blocking or affecting the users and how?
Mark them by the level of urgency based on the impact you have determined. Do this to the best of your abilities:

🚨: Critical (affects core functionality or many users)
⚠️: High (significant, but not blocking)
🟡: Medium (minor but recurring issues)
🔵: Low (cosmetic or non-urgent)
⁉️: Unknown (you are not sure)

Communicate by starting a thread for each error:

Share a summary of your findings to help others understand the context.
Give a brief summary of the error, if you know what it is.
Provide links to articles, resources, wikis, code where applicable.
Suggest an acceptable turn around time for the error to be resolved depending on what you have learned and the urgency of the error.

Focus on the highest-priority error that you can address and resolve it before moving to the next.
If you can't handle the error, allocate or escalate it but be the nagging kid in the toy store or that parent who's always probing about the homework. Follow up to ensure progress—it’s better to be persistent than to let issues fall through the cracks.

Investigate

When you are in a good position to get your hands dirty, use an emoji 🕵️ (like the detective one) to indicate that you have started. Document what you uncover. Add it to the thread. The more information you add, the better the outcome and aid for future dilemmas. If you are stuck, ask for help! Again, it's okay to not know everything.

By following these steps, you can systematically unravel even the most obscure errors and contribute to a more resilient system.

I'm going to outline what I typically do. These are not in a particular order and they are often revisited multiple times.

These steps relate to me currently using GCP (Google Cloud Platform), GKE (Google Kubernetes Engine), Error Reporting and the Logging modules.

Error Messages

Look for the error code, message, data identifier, raw stack trace and anything else that stands out.
Examine the samples.
See the occurrences, when they started, when the latest occurrence was.
See the pods that were affected.

Logs

Logs are a goldmine of real-time system information, whether the error is clear or obscure. There can be a lot of noise so filter out the nonsense.

Filter the logs by the pod name.
Filter the logs by log type - examples include verbose, info, warn, error - (sometimes I am unlucky with this) but it's a good starting point to choose warning or error and above over the default.
Hone in on a specific date and time range.
Search on specific keywords.

Search

Once you've mined the logs for clues, your next step is to search for broader context or existing solutions. Search through the rich history in your text communications channel like Slack, your wiki, emails, and the internet like Stack Overflow, GitHub issues, ask AI, etc. for the error message or stack trace. Give as much context and information as possible.

If you are really stuck, reach out on forums or other communities like on Slack or Discord.

Reproduce

Try to reproduce the error with the information you have gathered.
Do this in a lower environment, if possible (locally, development, staging, etc). This is often the fastest way to identify or confirm the root cause. It also protects the affected environment from further damage by not causing more chaos like with data corruption, downtime, warped analytics or user impact.

Code

Go to the code and see if you can identify the source of the error.
Recreate the conditions or data that triggered the error to understand its context. Analyze the section for logical errors, typos, edge cases, or potential misconfigurations.

If you don't have the stack trace or the error is not in the logs, you may need to add more logging to the code to get more information.

Document

Document your findings in the thread.
This is important for the next person who may experience something similar.

Include the root cause analysis, if you identified it.
Include the steps you took to investigate the error.
Include information that you found useful and what you found to be irrelevant.

Resolve or Escalate

Now it is time to either fix the error or give it to someone else who can–hopefully–fix it. The baton may be passed to another team member, but ensure clear handover and follow-up to maintain accountability.
Just ensure that the ownership is clear and that the error is not forgotten.

Resolve the Error

Whoo-hoo! You found the root cause, or a potential issue (sometimes multiples), and you know how to fix it, sometimes sort-of. Great job!

Apply the fix in the appropriate place to the best of your ability.
Ensure you have added some sort of test (or more) to confirm the error has been fixed as to mitigate it from breaking again in the future.
Test it thoroughly with the same steps you took to reproduce the error.
Use systems thinking to mitigate regression to other flows, users, applications, services and/or jobs. (Maximum effort!)
Get this fix tested by your colleagues or peers.
Deploy the fix to the environment where the error occurred.
Mark the error as resolved in GCP so that if the error occurs again, it will be a new error and you will be alerted.
Use an emoji ✅ in Slack to acknowledge that you have resolved the error and remove the other emojis.

Escalate the Error

So you got unlucky this time round and you can't resolve the error. It's okay. Mark the error in Slack as escalated using the emoji ⬆️, remove other emojis and allocate the error to someone else.

Try to be included in the investigation process by pairing with the person who picks it up, assuming you have capacity. Be a sponge to learn as much as you can from this experience and how the person tries to resolve the error. Your presence is beneficial because of your investigation and findings, the learning opportunity and the potential to contribute to the resolution. Ensure the issue is thoroughly documented and shared, fostering transparency and team learning.

Tips

By following these tips, you can build a resilient system and a strong, collaborative team.

Be brave: Don't be afraid to pick up errors. It's a great learning opportunity and you are not solely responsible for it.
Dig deep: Search for and identify the root cause of the problem. Don't just fix the symptoms. Look at the data, code, logs, and the environment.
Reduce the noise: Identify and reduce the noise in the system, such as overly verbose health checks or misclassified log levels.
Create visibility: A good mental model of your systems, data and code is beneficial to solving for errors so create tangible mind maps or documentation for the whole team to benefit from. Miro and Docusaurus are excellent tools for this.
Work together: Foster a culture of collaboration and sharing. You don't have to be the sole gatekeeper of errors. Create a round-robin system where everyone gets a chance to pick up errors. Hold regular error triage meetings or pair programming sessions to share insights.
Prioritize: Allocate specific blocks of time to address errors while reserving uninterrupted time for your regular tasks. Prioritize errors based on urgency and impact.
Process: Formulate a process that works for you and your team. It's important to have a process that is flexible and can be adapted as you learn more about the errors that are occurring.

Conclusion

Strong team habits are the foundation for addressing errors promptly and writing maintainable, supportable code.

Take ownership to strengthen your systems, support your team, and grow as a professional.
Write code that reports errors meaningfully, making debugging easier.
Communicate transparently with peers and leaders to secure the support you need and emphasize the impact of errors.

Embrace errors as opportunities to learn, grow, and build a stronger, more resilient system—and a more capable you. So, put your curiosity gloves on, flick fear on the ear and grab those errors no matter how obscure.

How to Reframe your Inner Imposter

Clarice Bouwer — Sun, 10 Nov 2024 17:02:55 +0000

Struggling with imposter syndrome? Find out mind-blowing tricks to flipping your self-doubt into unstoppable confidence!

This article delves into transforming imposter syndrome from a debilitating obstacle into a powerful motivator for personal growth. It suggests reframing feelings of self-doubt as natural signs of stretching one's abilities rather than indicators of inadequacy. By embracing vulnerability and recognizing that even successful individuals experience similar doubts, one can shift their mindset to view challenges as opportunities for learning. I offer practical strategies, such as positive self-talk and focusing on progress over perfection, to help readers turn their inner critic into a catalyst for confidence and development.

Read more about it at curiousprogrammer.dev - How to reframe your inner imposter.

Dynamically execute Tailwind CSS on multiple files with multiple outputs

Clarice Bouwer — Sun, 09 Jun 2024 16:00:50 +0000

Finds all CSS files in a given director, iterates through them and executes tailwindcss CLI using the Tailwind config and outputs each file to a dist/ dir.

find . -name \"*.css\" -exec npx tailwindcss -c tailwind.config.js -i {} -o dist/{} \\;

How to update outdated npm packages

Clarice Bouwer — Mon, 20 May 2024 12:59:46 +0000

Keeping your npm packages up-to-date is crucial for maintaining the security and performance of your projects. In this guide, I'll walk you through the steps to identify and update outdated npm packages.

The npm outdated command checks the registry for outdated installed packages. It shows direct dependencies by default, but with the --all flag, it includes all outdated meta-dependencies. Key output fields include wanted (maximum version satisfying semver in package.json), latest (latest version in the registry), location, and depended by. Red indicates a newer version matches semver requirements, and yellow indicates a newer version above semver requirements. Various flags like --json, --long, and --global modify the output format and scope.

For more details, visit npm outdated documentation.

Note that by default npm update will not update the semver values of direct dependencies in your project package.json. If you want to also update values in package.json you can run: npm update --save (or add the save=true option to a configuration file to make that the default behavior).

For more details, visit npm update documentation

How to write formatted EDN to a file in Clojure

Clarice Bouwer — Thu, 29 Feb 2024 07:40:12 +0000

You could be generating a config file, constructing a data file or downloading/extracting data from a data source. Sometimes, it needs to be human readable to make sense of it.

(spit "file.txt" (with-out-str (clojure.pprint data)))

(with-open [wr (clojure.java.io/writer "file.edn")]
  (.write wr (with-out-str (clojure.pprint/pprint data))))

How to get the nREPL port for a running REPL

Clarice Bouwer — Thu, 26 Oct 2023 04:47:44 +0000

I want to connect to a running REPL on my local machine outside of Visual Studio Code. The terminal output is cleared or truncated depending on the terminal.

There are two methods I know of.

Grep the processes for the name of the project that is running on the JVM. This will give a process ID. Eg. 71141

ps aux | grep name-of-the-project

Grep the network-related information such as open connections, open socket ports with the process ID from above.

netstat -vanp tcp | grep 71141

Find the loopback IP with the associate port. Eg. 63361

127.0.0.1.63361

Use that port in Visual Studio Code.

To find the port from the Clojure REPL: (slurp ".nrepl-port")

See: https://stackoverflow.com/questions/77364142/how-do-i-get-the-nrepl-port-for-a-running-repl

How to read all bytes from a resource file in Clojure

Clarice Bouwer — Wed, 16 Aug 2023 00:17:21 +0000

(-> "images/something.jpg"
     io/resource
     io/file
     .toPath
     java.nio.file.Files/readAllBytes)

It is important to note that the resource is located in your resources directory so use a path relative to your application or components root directory.

How to configure python environs for Visual Studio Code

Clarice Bouwer — Sat, 29 Jul 2023 12:00:19 +0000

If you need to store environment variables for Python that does not automatically get detected by VS Code then this answer should hopefully help you.

I am doing a Data Science bootcamp and need to store a specific path for one of my challenges.

Exporting PYTHONPATH in ~/.zshrc works perfectly when I run the notebook directly from Juypter, however the environment variable is not detected through VS Code.

I created a global file in my home directory from the pwd (present working directory) that I wanted to store in my path:

echo "export PYTHONPATH=$(pwd):$PYTHONPATH" >> ~/.python-env

I opened the JSON User Settings in VS Code and added the python.envFile setting which points directly to ~/.python-env

{
    "python.envFile": "~/.python-env"
}

Python Data Science Cheat Sheets (a WIP project)

Clarice Bouwer — Fri, 14 Jul 2023 03:48:39 +0000

I am doing a Data Science bootcamp through Le Wagon. I have found static cheat sheets that I have turned into interactive versions using Jupyter Notebooks so that I can better understand the packages available for use.

I might be re-inventing the wheel but this approach is helping me reinforce the knowledge and properly explore the packages.

If you are interested, check it out, star it, fork it, clone it, contribute to it, share it.

It's a work in progress project. From one package starting with numpy, I am hoping to grow it into numerous packages that make sense.

If you have any thoughts, suggestions or words of encouragement 😉, please let me know.

Made with ❤️ for Data Science 🔬 🧮 and Python 🐍

How to create a string from values joining with commas and an and.

Clarice Bouwer — Mon, 03 Jul 2023 09:49:08 +0000

I have a vector of errors that looks like this:

(def messages 
  [{:error "Name is required"}
   {:error "Last name is required"}
   {:error "Date of birth is in the wrong format"}])

(->> messages
  (map :error)
  (remove string/blank?)
  (string/join ", ")
  (#(string/replace % #", ([^,]+)$" " and $1")))

The above form will output the following result:

"Name is required, Last name is required and Date of birth is in the wrong format"

It uses a threaded macro to map through all :error keywords, removing blanks (and nils), joining by comma and then replacing the last comma with an and to read nicely.

This gives me a comma-separated list

How to import with absolute paths in Gatsby

Clarice Bouwer — Fri, 30 Jun 2023 02:34:04 +0000

gatsby-node.js:

const path = require('path')
exports.onCreateWebpackConfig = ({ actions }) => {
  actions.setWebpackConfig({
    resolve: {
      alias: {
        components: path.resolve(__dirname, 'src/components'),
        templates: path.resolve(__dirname, 'src/templates'),
      },
    },
  })
};

./project/src/path/to/some/file/of/yours.js:

import Layout from 'components';

as opposed to:

import Layout from '../../../../components/Layout.js';

See GitHub solution as original source and reference.

How to get distinct values in a grid

Clarice Bouwer — Thu, 29 Jun 2023 19:20:02 +0000

def get_distinct_values(values):
    return list(set(filter(lambda x: x != 0, values)))

This code will get a distinct list where values are not equal to zero. I used this code in a Sudoku solver algorithm.