DEV Community: Azhar Momin

My LFX Mentorship at OpenTelemetry Has Come to an End

Azhar Momin — Sun, 06 Sep 2026 18:30:00 +0000

My LFX Mentorship at OpenTelemetry Go Compile Instrumentation (otelc) has ended. I’ve already written about how I got here and why I applied and about the technical work, so this one’s just a short summary and reflection.

What shipped

The biggest piece of work was import-driven instrumentation selection, covered in my last post.

The other piece, registry-backed integration search through the OpenTelemetry Ecosystem Explorer, is still in progress. We’re also migrating otelc’s instrumentations into a separate contrib repo, which should make this kind of work easier to build on. I’ll write about these in a future post.

What I gained

Getting better at communication was one of the reasons I applied. I was always fine talking async over GitHub, comments, reviews, PR descriptions, but I didn’t have much confidence in live SIG meetings or calls. Weekly SIG meetings and regular syncs with my mentors fixed that.

I’m also a lot more interested in observability than when I started. I want to keep exploring the OpenTelemetry ecosystem, mainly around Rust and eBPF. I’m graduating in about a year, and this mentorship is a big part of why I’m now seriously looking at observability as a career direction.

The project got a lot of traction during my mentorship, we submitted two project proposals for the next LFX term, so alongside my own contributions I reviewed many PRs. Reading other people’s approaches and review discussions taught me things faster than writing my own code ever did.

What’s next

I’m mentoring a project myself next term, co-mentoring with Xabier Martinez. I’m also planning to stay in touch with my mentors and everyone else I’ve gotten to know through this.

Thank you

Thanks to my mentors, and to the OpenTelemetry and CNCF community, for everything over the past few months. ❤️

Building Import-Driven Instrumentation for otelc

Azhar Momin — Wed, 12 Aug 2026 06:30:00 +0000

In my last post, I wrote about how I got into OpenTelemetry Go Compile Instrumentation and applied to LFX Mentorship. This one is about the actual work, what I built, the bugs I ran into, and how I fixed them.

Why otelc specifically?

I’m interested in both low-level tooling and observability, and otelc sits right at the intersection of the two. That’s really what pulled me toward this project instead of OpenTelemetry in general.

What is otelc, briefly?

otelc instruments Go code at compile time. Instead of requiring developers to manually instrument their code, otelc rewrites the program during compilation so that instrumentation is inserted automatically. The rest of this post assumes a little bit of that context, so here’s the short version before I get into what I actually worked on.

What I worked on

Import-driven instrumentation selection

otelc ships with a built-in set of instrumentations, each made up of modules and rule files. But there was no good way to bring in third-party instrumentations easily, and no way to disable a built-in instrumentation at compile time (you could only toggle them at runtime).

This came up in a SIG meeting, and was tracked in this issue. Orchestrion, Datadog’s own compile-time instrumentation tool for Go, already had a solution: an orchestrion.tool.go file that developers could use to declare instrumentation explicitly. We used that as a starting point and built something similar for otelc.

The mechanism works like this: otelc looks for a file (otel.instrumentation.go, or otelc.tool.go as an alias) containing a list of blank imports. This file isn’t included in the actual build (it’s marked with a tools build tag, following the standard tools.go convention), but Go still tracks its imports in go.mod. otelc reads this file before building, loads the imported packages using golang.org/x/tools/go/packages, and scans each package’s directory for rule files (otelc.yaml, *.otelc.yaml) or another otel.instrumentation.go, allowing instrumentation packages to bundle together other instrumentation packages. Those rule files define how to instrument that package.

Here’s roughly what one of these files looks like in practice:

//go:build tools

package tools

import (
 _ "go.opentelemetry.io/otelc/instrumentation/net/http/client"
 _ "..."
)

The naming itself was already decided in a UX design document from before I joined the project, so that part was settled. I’d originally written this as a single PR, but it ended up being around 5,000 lines, which was far too large to review properly. Reviewers asked me to split it into smaller PRs, so I did. During that process, one review comment asked for an integration test covering a recursive instrumentation path, where one otel.instrumentation.go file references another. Writing that test surfaced a real bug: otelc was only looking for hook code files inside its own embedded bundle, not in packages loaded from external import paths. If that comment hadn’t come up, this bug might have gone unnoticed for a while. I fixed it by updating otelc to load hook code through golang.org/x/tools/go/packages as well. The implementation landed across a couple of PRs: the hook loading fix in #617, followed by the core import-driven instrumentation support in #612.

The initInstrumentation deadlock

Every instrumentation in otelc followed the same rough pattern: lazily set up the OTel SDK (if it wasn’t already running) and start runtime metrics, guarded by a sync.Once.

This caused a deadlock. Here’s what happened: when a user’s code called grpc.NewClient, the gRPC instrumentation hook would call initInstrumentation, which runs inside a sync.Once.Do. If the user had their OTLP exporter configured to use gRPC, the SDK setup inside that same Once.Do would itself call grpc.NewClient, which tried to call initInstrumentation again, re-entering the same Once.Do block that hadn’t finished running yet. That’s a deadlock.

My first fix was simpler than it should have been: check the value of OTEL_EXPORTER_OTLP_ENDPOINT and skip instrumentation for any grpc.NewClient call pointed at that endpoint. That worked for the obvious case, but it broke down as soon as the two values didn’t match exactly, since the values could point at the same collector without matching as strings at all:

endpoint = "http://collector-localhost:4317"
target = "localhost:4317"

This kept happening because I was relying on dumb string comparison to detect whether the two calls pointed at the same target. Instead of trying to make that comparison smarter, I took a different approach.

The fix was to stop having every instrumentation independently lazy-init the SDK. Instead, I moved that startup logic out into a single injected rule that runs once in the user’s main package, before any instrumentation hook runs. Every hook can now assume the SDK is already set up by the time it runs, so there’s no endpoint comparison to get wrong in the first place. It also meant third-party rules don’t need to follow the old init pattern at all, they can simply assume the SDK is already available, which fit nicely with the import-driven instrumentation work happening around the same time. Users who want to bring their own tracer or meter provider can also remove this rule and wire up their own. The implementation landed in #625.

otelc pin and auto-pinning

The second thing I worked on was the otelc pin command, which generates an otel.instrumentation.go file based on a user’s dependency graph, inspired by Orchestrion’s own pin command. Orchestrion’s version points to a single package that bundles every integration together. I changed this so otelc pin looks at the user’s actual dependencies and infers which instrumentations are relevant, rather than pulling in everything.

otelc pin also keeps the generated file clean by automatically removing imports that don’t resolve to any rule files (or other otel.instrumentation.go files). The result is an explicit list of the instrumentations that are actually relevant to the project.

Auto-pinning was also mentioned in the original issue, but initially I wasn’t convinced it was necessary. If a project didn’t have an otel.instrumentation.go file, otelc could simply fall back to its default instrumentations. Generating a temporary file just to represent that default set seemed like an unnecessary extra step.

While testing otelc against a project using an older version of gRPC, I ran into another problem. I already had a fix open for that problem, but auto-pinning addressed its underlying cause in a much cleaner way.

At the time, otelc’s build process matched instrumentation rules first, then added the matched instrumentation modules to go.mod and ran go mod tidy. That dependency resolution could bump the project’s gRPC version. The instrumentation rules had already been selected against the old version, though, so otelc could end up building against a dependency graph different from the one it had used to decide which rules to apply which resulted in the build failing.

My original fix was to run the setup process in multiple passes until the dependency graph stopped changing. It worked, but auto-pinning turned out to solve the same problem much more cleanly.

otelc go build now generates a temporary otel.instrumentation.go early when the user hasn’t provided one. Dependencies are resolved and added to go.mod as part of that pinning phase, before rule matching happens. By the time otelc selects instrumentation rules, it’s looking at the dependency versions that will actually be used for the build. The setup phase that follows no longer needs to modify the dependency graph, so the selected rules stay valid.

This also gives us a cleaner path toward registry-backed integration search. If otelc eventually discovers compatible instrumentations from the OpenTelemetry Ecosystem Explorer registry, the same pinning step can resolve those integrations and their dependencies before rule matching, rather than requiring the rest of the build pipeline to care where an instrumentation came from. I talk more about that idea in the registry-backed integration section below.

One rough edge that’s still open is that otelc pin currently adds replace directives pointing to local paths in go.mod, which makes checking the generated file into version control awkward. Decoupling instrumentations from the tool itself, also described below, should fix that. The initial implementation landed in #655.

What’s next

A few things I’m looking at for the rest of the mentorship:

Decoupling instrumentations from otelc’s embedded bundle. Right now, instrumentation packages are embedded directly into the otelc binary. That means adding a new instrumentation requires a new release of the tool, even if it’s otherwise fully compatible. Decoupling this would also let us build a proper dependency graph between instrumentations (for example, automatically pulling in a runtime/GLS instrumentation whenever the OTel SDK instrumentation is used), and would let users check reproducible otel.instrumentation.go files into their own repos without local path replaces.
Registry-backed integration search. Instead of only looking at the embedded bundle, otelc could look up compatible instrumentations through the OpenTelemetry Ecosystem Explorer registry, which would let third-party instrumentations get picked up automatically once published there. This is currently being researched, see this issue for the open questions around how Go compile-time instrumentations should be represented there and how otelc would consume that data.
Supporting .otel.yml files. Similar in spirit to otel.instrumentation.go, but without tracking dependencies in go.mod. Useful for customizing instrumentations without wanting them tracked in version control, though it won’t produce fully reproducible builds.

Acknowledgements

The work described here went through a lot of review. Thanks to Kemal Akkoyun for raising the original issue that led to import-driven instrumentation selection, and to both Kemal and Dario Castañé for reviewing the PRs linked throughout this post, including the review comment that led me to the recursive-instrumentation bug. Thanks also to everyone else in the OpenTelemetry Go Compile Instrumentation community who reviewed and discussed this work along the way.

My Path to LFX Mentorship, and Some Advice for Future Mentees

Azhar Momin — Fri, 24 Jul 2026 00:00:00 +0000

How I got here

I was building a Discord minigame as a side project in Python. It grew bigger than I expected, and it started slowing down, partly due to code I could’ve written better. Eventually I decided to rewrite it in Go, mostly to see how much its concurrency model would help.

Separately, I realized that whatever language I ended up in, I wanted real visibility into what the bot was actually doing, not just something that told me when it crashed. So I tried instrumenting things manually, and ran into two problems almost immediately. First, I didn’t really know what to instrument. Second, even following the docs, there was so much boilerplate involved that I nearly gave up on the whole idea.

That’s what pushed me toward auto-instrumentation, mostly as a way to avoid the boilerplate. At the time, though, I was more interested in learning how to instrument things properly than in auto-instrumentation itself, that appreciation came later.

While looking into zero-code instrumentation solutions, I came across this old blog post announcing the new OpenTelemetry Go Compile Instrumentation project. One of the most appealing reasons for me to contribute was that there was scope for adding new instrumentations, a learning chance for me since I didn’t know much about observability yet. The compiler-level part also mattered to me, I already liked that kind of low-level work. On top of that, I’d separately been looking for a CNCF project to contribute to. I introduced myself in the project’s Slack, asked some questions, and my first real contribution was auto-instrumentation for k8s.io/client-go, though there were a couple of smaller ones before that.

Before this: GSoC @ KDE

This wasn’t my first mentorship. In 2025, I was a GSoC mentee at KDE, working on integrating KDE libraries into OSS-Fuzz under the guidance of Albert Astals Cid. At the time, I was a total newbie to open source. That period taught me the ropes, both technically and in terms of how contributing to a real project actually works. I’ve written about that in more detail separately (link), and it’s worth mentioning here because it’s a big part of why I felt ready to jump into OTel later.

Why Mentorship again?

Two reasons, really.

The first was that I wanted to get better at communicating. LFX comes with weekly SIG meetings and regular meetings with my mentors, Kemal Akkoyun and Dario Castañé. Having to regularly explain what I’ve been working on to people who know the project well has been good practice, especially since it isn’t something that comes naturally to me.

The second was that I wanted to go deeper into observability and OpenTelemetry specifically, and get better at Go along the way. We’re still in the middle of the mentorship, but so far it’s been exactly what I was hoping for.

Advice for future mentees

A few things that I think mattered, based on my own experience going through this.

Contributing is more than opening PRs. Looking back at what actually helped my application, joining community discussions, opening issues when I found something worth raising, reviewing other people’s PRs, and sharing ideas even when I wasn’t sure they’d go anywhere all mattered just as much as the code I wrote. Mentors and maintainers see all of that, not just your commit history.

Reviewing other people’s PRs ended up helping me far more than I expected. Reading different approaches, following review discussions, and seeing why maintainers asked for certain changes taught me things I wouldn’t have learned nearly as quickly by only writing my own code.

Consistency also matters more than volume. You don’t need dozens of PRs to stand out. Showing up regularly, staying involved in discussions, attending meetings when you can, and gradually becoming part of the community leaves a much stronger impression than trying to maximize your contribution count over a few weeks.

Be genuine about why you’re there. From what I’ve seen, it’s fairly easy for mentors and maintainers to tell the difference between someone who’s actually excited about a project and someone who’s mainly interested in the resume line. You can want both, a career boost and genuine interest at the same time, but if the interest isn’t real, it tends to show once the work gets less glamorous.

Don’t disappear if you’re not selected. LFX mentorship slots are limited, not getting selected doesn’t say anything about your ability. If you were seriously contributing during the application period, you already learned something real, keep using it. That doesn’t stop mattering just because one specific mentorship round didn’t go your way. You can keep contributing to the project on your own, and there’s always a next round.

Don’t hesitate to ask. Whether it’s a basic question about the codebase, something you think might be obvious, or just wanting to double check you’re on the right track, ask. I did this myself when I first joined the project’s Slack, and no one made me feel bad about it. Most maintainers would rather answer a simple question early than review a PR that went in the wrong direction because you guessed instead of asking.

On using AI tools. I do use AI tools regularly, but where and how varies. For code, I use it sometimes, mostly for parts I already understand well, where I’m confident I could write it myself but don’t need to grind through it by hand to learn something new. Where I use it a lot more is research and understanding things, I reach for an LLM over Google search these days because it explains things faster than digging through search results and docs myself.

Wherever I use it, I double check everything it gives me. I review answers against the actual source, and I review AI-assisted code multiple times before submitting it.

Reviewers are volunteering their time, so submitting unreviewed AI-generated code doesn’t just leave a bad impression, it’s disrespectful. Make sure you’re following whatever AI usage guidelines the project you’re contributing to has. Also question AI-generated code critically: ask why something was done a certain way, whether it’s actually needed, read through the surrounding code, and check if there’s a simpler solution.

More generally, whether code comes from AI, Stack Overflow, or another contributor, you’re the one submitting it. Make sure you understand it well enough to explain every part of it during review.

Closing thoughts

This mentorship also convinced me I want to be on the other side of it. I’ll be mentoring a project myself in the next LFX term, co-mentoring with Xabier Martinez. You can check out the project here. I’m looking forward to it, partly because I want to try being the kind of mentor Kemal, Dario, and Albert have been for me. I’ll write more about the mentorship itself once it wraps up, and separately about the actual technical work I did on otelc.

Acknowledgements

Thanks to my mentors, Kemal Akkoyun and Dario Castañé, for their guidance so far during the mentorship. I’d also like to thank Xabier Martinez, who was one of the first maintainers I interacted with on the project, along with everyone else in the OpenTelemetry Go Compile Instrumentation community for the reviews, discussions, and feedback along the way.