DEV Community: alrex

Building the OpenTelemetry Collector

alrex — Mon, 21 Aug 2023 00:00:18 +0000

The OpenTelemetry Collector Builder is a tool that allows users to compile their own custom instance of the Collector. This is useful in a few different scenarios. Maybe the existing distributions contain more components than you need. Choosing specific components reduces the complexity and size of the resulting Collector. In my experience, the fewer components you’re including in binaries shipped around production, the happier the security teams tend to be.

Another reason you may need to use the Builder is that you’re producing custom Collector components. With the Builder, adding a component requires as little as a single line in the manifest file, and you can add a component hosted anywhere. Whatever your reason is, getting familiar with the Builder is a good practice if you’re in the business of deploying the Collector in your environment.

Using the Builder

One thing I’ve been wanting to make easier still is the process of using the Builder. Today, if you’re looking at building a local instance of the collector, installing the Builder and using it can be done in a few steps:

Install the Builder
Create a manifest.yaml file
Run the builder with a -config manifest.yaml flag
Profit!

Of course, building binaries on a local machine, then shipping it off to a production environment isn’t often a recommended practice. The better way to go about it is to put your configuration in a repository, and use a build pipeline to do all the work. This will ensure repeatability (usually…) of the build, along with keeping a record of the changes to your build. Unfortunately, having built a few different custom Collectors has caused me to copy and paste a lot of unnecessary code around. Could a custom action do it instead? It sure can!

Collector Builder GitHub Action

In order to reduce the amount of required boiler plate code, I wrote a custom GitHub action runs a Docker container with the Builder already installed. A user of this action specifies the path of the manifest file to use as configuration via the manifest-file input. The action will then produce a custom Collector binary! The following shows an example GitHub action configuration that clones the current repository, uses the collector-builder-action, and uploads the resulting Collector binary:

on: [push]

jobs:
  custom-collector-action:
    runs-on: ubuntu-latest
    name: A job to build a custom OpenTelemetry Collector
    steps:
      - uses: actions/checkout@v3

      - name: Custom Collector Step
        uses: codeboten/collector-builder-action@v1

      - uses: actions/upload-artifact@v3
        with:
          name: otelcolaction
          path: _build/otelcolaction

In my test repository, the manifest looks like any other Collector manifest file:

dist:
  module: github.com/open-telemetry/opentelemetry-demo/src/otelcollector
  name: otelcol-demo
  description: OpenTelemetry Collector for OpenTelemetry Demo
  version: 0.81.0
  output_path: ./_build
  otelcol_version: 0.81.0

receivers:
  - gomod: go.opentelemetry.io/collector/receiver/otlpreceiver v0.81.0
exporters:
  - gomod: go.opentelemetry.io/collector/exporter/loggingexporter v0.81.0
  - gomod: go.opentelemetry.io/collector/exporter/otlpexporter v0.81.0
  - gomod: go.opentelemetry.io/collector/exporter/otlphttpexporter v0.81.0
processors:
  - gomod: go.opentelemetry.io/collector/processor/batchprocessor v0.81.0
connectors:
  - gomod: github.com/open-telemetry/opentelemetry-collector-contrib/connector/spanmetricsconnector v0.81.0

And that’s all you need in your repository. Once the GitHub action is run, the Collector Builder Action should produce an artifact as it did in my case.

So what next? Well, the initial release of this action is full of sharp edges. There’s no support for multiple architectures. It would be great if this action could produce a Docker image. And I’m pretty sure there are many more things I’d like to improve. If this is useful to you, please let me know via whatever social platform. You can also open an issue or a pull request! And who knows, maybe this will eventually be donated to the OpenTelemetry project. In the meantime, enjoy building your Collector, now with even fewer steps!

Cover Photo by Marcel Strauß on Unsplash

Originally published at https://words.boten.ca on August 21, 2023.

Disabling go modules in Visual Studio Code

alrex — Tue, 26 May 2020 04:00:03 +0000

Photo by Jeff James on Unsplash

I couldn’t find a good answer to this question and I looked for hours. Ultimately, I decided to document it here for the next time I need it and for anyone else out there who’s looking to disable go modules in in Visual Studio Code.

Open up Settings (⌘,) and search for “go tools env vars”

Click “Edit in settings.json” and set GO111MODULE to off

I also disabled the go language server as it was burning a hole through my lap:

Well that’s all there was to it! I’m now able to use VSCode for all my editing needs.

Tests, delivery and other important things in software development

alrex — Tue, 01 Oct 2019 04:28:57 +0000

Tests, delivery and other important things in software development

There’s a scene from the Simpson episode “Lisa the Simpson” where Lisa appears on a television broadcast to make an appeal to the audience. She urges citizens to use and treasure their brain, in fear that she was about to lose her own due to the “Simpson gene”. As I was leaving my previous employer, I had a similar fear that my brain would turn to mush. I’m writing this article in the event that my true destiny of turning into a couch potato becomes fulfilled and I no longer have full use of my brain. The advice below is a little all over the map, but all of it has helped teams I worked with be more effective.

Credit: 20th Century Fox

Treat your Continuous Integration Continuous Delivery tooling like production

When starting a project, it’s easy to get caught up in building software and getting it deployed somewhere. Often in that excitement, someone says “Let’s get this into CICD”, at which point, someone else will volunteer to set the thing they know about up somewhere (ie. Jenkins, concourse, drone). As soon as the software is building and deploying, boom, let’s move on to the next thing. This is a key point in the project. It’s easy to say, in the name of building more features, this works for now, it’s fine. However, this is also the point in time at which we’ll look back to, on the day where something important MUST be shipped and we can’t because Jenkins is caught in an unrecoverable state, or the concourse workers have died and for some inexplicable reason, restarting them doesn’t fix the problem. There’s no need to spend days and weeks making CICD completely bulletproof, but some simple steps can ensure even a catastrophic failure doesn’t cause the development team pain down the road.

check-in configuration

One of the easiest ways to recover something, and also ensure it passes the bus factor test, check it in! Whether choosing to check-in configuration for the host, or create images to redeploy hosts easily, there’s more than one way to accomplish this.

centralize logging

No one looks at logs, that is until, something goes wrong. So it’s problematic when those logs are no longer accessible. The easy solution here is to ship the logs somewhere, anywhere. Use a centralized syslog system or some third party service. Ideally, the logs are shipped somewhere that makes monitoring and alerting on them easy.

alerting

Use your own discretion here, but some level of advanced warning when things are about to go BOOM is nice. We used to alert on Jenkins 24/7, and most of the time, these alerts could have waited until regular business hours. Some of the metrics we monitored were the usual suspects: disk, memory, CPU. Some other metrics that are handy to watch for is build times for known components.

document setup and recovery procedure

Anytime I join a team, I get to be the annoying person that asks questions about where things are documented. Often times, the thing that’s least documented is how someone put together the deployment pipeline and what to do when things go wrong with it. Document it, and test out the documentation.

apply chaos engineering practices

If you’re not familiar with chaos engineering, here’s a great website to get you started. In short, it means to break systems intentionally to figure out how to make them more resilient. I’ve worked in places that host break-a-thons or fire drills to achieve similar goals. In the case of a CICD system, I would use chaos engineering principles to test that the recovery procedure works, and that no significant data loss occurs in the case of a failure of the system.

backup and store artifacts separately

Losing the CICD is bad. Losing all the artifacts the company has shipped because there was only one copy and it lived on the CICD host can be disastrous. Yes, this has happened to me and no it wasn’t all that long ago. It was not a good week, but thankfully it was early enough in the project that we hadn’t shipped anything yet. An easy option for storing artifacts is to ship them to an online storage solution like Amazon’s S3. Concourse provides a custom resource for storing things in S3, and Jenkins provides this functionality via a plugin.

don’t maintain it

Unless an organization’s bread and butter is providing a CICD solution, it’s likely that the build pipeline is just another required tool. Don’t waste engineering time on it, if it’s in the budget, use a hosted solution.

Emit events when deploying builds

Tying in deployments with changes in the code is awesome. I like to hypothesize about the impact of a change. One of my favourite feelings is that sense of accomplishment that comes from seeing it being deployed and observing its impact on a dashboard. Being able to correlate changes in the behaviour of a system with deployments is super duper helpful when troubleshooting what went wrong. Key information I’ve found useful in the past:

commit and link to change in source control
link to build logs
start and end of deployment
phase, if applicable

Write tests

I’m not about to preach about test coverage, or unit testing all the things everywhere all the time. I know that’s not practical in all cases, and in some cases, the value add of the unit tests would be close to none (ie. when the test mocks 95% of the functionality and the rest of the code is just shim around what’s being mocked). But writing tests when starting to tackle a piece of logic that is tricky, or even a piece of logic that appears trivial but requires a fair amount of code, almost always pays off. And not only does it provide coverage for regressions and more safety for other engineers, but it also feels great. I don’t know, when I write tests and I finish shipping the code, it just makes me feel like I’ve done my job right. When writing tests, here’s a couple of things to keep in mind.

make it easy

There’s a ton of tools and frameworks out there to provide additional functionality for folks writing unit tests. My general rule of thumbs here is that tools used in unit tests should not make writing and running tests harder for folks on the team. Adding barriers to writing tests will ultimately drive developers away from writing them. It’s important to keep things as simple as possible.

test what makes sense

A lot has been documented about code coverage and about quality of code based on that coverage. I’m a big fan of testing what makes sense in a codebase. It’s easy to get a false sense of security about the codebase based on the amount of the code that is covered. It’s also easy to get into the trap of writing so many tests, that writing a line of code means modifying dozens of tests in a more or less meaningless way.

Pair with people

Take the time to pair with people. No matter how important delivering a particular feature seems, ensuring others are empowered to take over whatever task at any given time is precious. This works both for the person learning and the person offloading their knowledge. In my last few weeks at my previous job, I took every opportunity I had to ensure everyone around me would know what I was working on. I’ll admit, sometimes it felt like uncovering a pile of dirt I had swept under the rug, but somehow, I still felt better after doing it.

it feels scary

It’s important to acknowledge that not everyone feels comfortable pairing. Different personalities, experience levels and biases based on previous experiences with pairing will all play a role in it. Even after years of pairing, occasionally a spike in adrenaline happens when someone asks me to pair. Imposter syndrome is real, and there’s always a chance to be found out when pairing.

it’s intense

Pairing can be quite intense. Writing code while continuously working with someone else on what is next can be quite taxing. It’s important to remember to take breaks, and to limit the amount of pairing done daily.

different tools

Tooling can be a real barrier when pairing. Pairing on someone’s laptop usually means the person whose laptop is being used will do all the driving, or that the other person will keep asking how to do things. This leads to an increase in frustration and eventual abandonment of the process. The best setup I’ve ever had the chance to pair included dedicated pairing stations in the office. These were separate from individual workstations. Each pairing station was setup identically, with an image that was used to wipe and reset the stations regularly. This meant no one had any attachment to the pairing station, and all the tools that were used were installed as part of the base image. Another good way to pair has been through tmux shared sessions or using the Live Share extension Visual Studio Code’s which allows everyone pairing to collaborate equally. A last resort method of pairing is using screen sharing through video conferencing software. The downside of this approach is one person is driving, while the other is passively involved.

Credit: https://xkcd.com/1513/

Most of what I covered here can be experimented with cheaply in an organization. Starting with a couple of hours on a Friday and experimenting, seeing what works and what doesn’t. If something doesn’t work after having giving it an honest try, throw it away, and send me some feedback! I would love to learn from your experiences.

The journey back to being an individual contributor

alrex — Thu, 03 Jan 2019 06:24:18 +0000

A few people have asked me about my recent job title change. After managing an infrastructure team for two years, I switched career tracks back to an individual contribute role. I wanted to share some thoughts about the transition in hope that it might help others in similar situations.

The journey

A few months ago, I was in one of those deep pits of self doubt. You know, the ones where you question everything you say and do. I was getting to the point where I was really starting to question whether or not I was still able to write code. In my first year as a manager, I was able to scrounge time outside of work to write code. But with toddler and a newborn at home, all my free time was spent parenting.

Around the same time, I began reading The Seven Habits of Highly Effective People. Many had recommended the book to me over the years, but I’d just never gotten around to reading it. The book drove me to start looking inwards at what I could control to increase my level of happiness. What did I have the power to change in my current situation? I wanted to write code to increase my happiness, but could not find the time to do it. So I looked for ways to cut out any time-wasting activities that were still in my daily routine. This allowed me to gain back little bits of time that I set aside specifically for programming. Soon I was getting so excited about writing minuscule amounts of code that I would stay up late into the night, waking up the next morning full of the excitement of getting something working the night before.

This was the key turning point. One morning, I started telling my wife all about the mostly irrelevant piece of code I figured out the night before. She made a comment along the lines of not having seen me this excited in years. That’s when I knew something had to change.

The myth

Some believe that moving to management is a promotion. That after being an individual contributor for over ten years it’s the only choice in order to move up. For a long time I believed this as well. In some industries, the management track is directly tied to financial compensation. The more people you’re responsible for, the more money you make. In the software industry, once you reach a certain level in your career, it becomes a set of parallel tracks. Becoming a manager isn’t about moving up in a career as much as it is moving in a different direction altogether. Instead of writing and designing solutions with code, you’re solving problems from a different dimension.

This myth caused me to spend weeks agonizing on whether or not I should have the discussion with my manager about wanting to change track. Was I taking a step back in my career? After all, I was managing up to eleven people in multiple projects at some point. Isn’t that how success is measured? Isn’t saying that I want to write code again like giving up?

Motivation

One of the most common questions I get asked is about what motivated me to switch tracks. Looking back now I realize that there were a few things I know about myself that I didn’t before:

I love writing code. Working with technology and exploring new problem spaces drives me. Enabling others via proof of concepts and building things gets me excited.
I measure my success on what I personally accomplish. As a manager, I viewed my job as enabling my team to succeed. When the team achieved success, I was excited for the members of my team. As folks grew and tackled bigger responsibilities, I was excited for them, but I never really felt a sense of accomplishment from it.
I’m driven to create impact. After many talks with my partner, I was convinced that I could have a greater impact for my organization by stepping away from my current role.

Learnings

Though I decided to step off the management track, I’m happy to have had the opportunity to manage a great team. I learned a ton while being a manager. Some of these are things people told me early on, but didn’t quite register until later. Some might seem obvious, but they’re worth re-iterating.

Stop worrying about writing code. The sooner you can let go of the problems you were solving, the sooner you can start thinking about the new problems you must solve.
Reach out to your new peers for help. All managers were new at this at some point. They understand the challenges of the role and can share valuable experiences.
Give responsibilities to people on your team. It provides them a chance to step up, and you a chance to free up some of your time to look ahead. Owning a problem, product or project doesn’t mean you have to do everything yourself.
Find a way to measure your input and output for the week that motivates you.
Find a good coach. I was lucky enough to have access to some great coaches that helped me work through challenges I faced early on.

The future

Folks ask me whether or not I am giving up on management altogether. The answer is I don’t know. What I do know is that at this point in my career, it’s not right for me. I’m happy to write code and believe this is where I can have the most impact at this time.