DEV Community: Leon Adato

Knowing What’s Under the Hood Helps

Leon Adato — Mon, 27 Jul 2026 14:00:00 +0000

I’m not a car guy. I can drive them, but I don’t know the first thing about fixing them, and not much about maintaining them. My son, on the other hand – who’s getting a degree in mechanical engineering – is most definitely a car guy. He’s learning a lot about cars, and he’s teaching me a lot about what I’ve been doing wrong, even with simple things like maintenance.

I was thinking a lot about that as chapter 2 of my O’Reilly book, titled “Wrangling Telemetry” was being written. What book, you ask?

A long, long time ago (it was April 21) in a blog post far, far away (you can find it here) I wrote about the O’Reilly book that Avi Freedman and I are writing about Network Observability.

The book title is “Practical Guide to Modern Networking Telemetry”. The raw, “lightly edited”, somewhat unexpurgated version of chapter 2 is available today.

You can find it here: https://www.kentik.com/go/ebook/oreilly-guide-to-modern-networking-telemetry/.

I will briefly point out that it’s REALLY worth your time to download this copy because the final version will be about 120-150 pages (those O’Reilly editors don’t work for free!), but the raw version clocks in at 82 pages, and that’s just up to the end of chapter 2. So you really are getting a whole lot of extra content right now – content that will be lost to what looks to be a machete-and-chainsaw editing process later this year.

But if those 80+ pages aren’t enough for you – if you want a little more behind-the-scenes extra content to close our your week – I wanted to expand on just one idea that I throw out there in the first couple of pages of chapter 2.

On page 42 I wrote:

“Yes, it’s true that vendor solutions do a lot of heavy lifting behind the scenes. But this book is about exposing and elaborating on what that behind-the-scenes activity is. Part of the reason is so you can be a more informed customer. Knowing what happens inside the black box of observability solutions, HOW you get from “data comes in the left side” to “pretty charts come out the right side” allows you to make educated decisions about everything from OpenTelemetry methodology to aggregation options to pipeline management.

No matter how well the amazing folks at O’Reilly edit down chapter 2, the question remains: “Why should I read an entire chapter about how the observability sausage is made? Why do I care about enrichment, normalization, pipelines, data storage, and the rest. Like Avi and Leon SAY, RIGHT THERE IN THE BOOK, the vendor handles that shi…tuff.”

Let’s go back to my car analogy. It seems I’ve been spending a lot – too much – on oil changes. I thought that natural oil was better than synthetic. It was also significantly cheaper. With his experience on two different teams that compete in Baja SAE he’s learned a lot. And – without getting into the nitty gritty (because I still don’t understand all of it, and this is a blog about observability not oil) the synthetic oil is better for the cars we have, and will last longer, and overall work better.

That’s information I didn’t have because I didn’t choose to think about how my car operated on more than a surface level. I was content to let the vendor (the quicky-oil-change place) make my choice for me, and to make decisions based on price rather than effectiveness.

I find – especially when it comes to monitoring and observability – that IT practitioners can fall into the same trap. We already have too much on our plate. We already carry a huge cognitive load. Learning about the internal process of a tool that – let’s be brutally honest for a moment – is often seen as little more than something we do to satisfy an audit doesn’t seem like a good use of our time.

But – like that oil change analogy – it can have massive long-term impact. Understanding how your monitoring tools “wrangle” the data can

help you avoid cost overruns
understand why data doesn’t always match (what you expect or what other tools are telling you)
use your tools both effectively and for the right use cases
avoid wasting money replacing tools “that suck” when in fact they don’t suck, you’ve just been using them incorrectly.
and more.

With all of that said, I don’t believe that knowing how to wrangle your telemetry is as useless as some people think. With the rise of Open Telemetry, those pipelines – and all the techniques to normalize, filter, sample, and transport data that they perform – is becoming more important than ever to understand.

So consider this chapter – in it’s full, lightly edited, unredacted form – to be Avi and my gift to you to become a better user of your monitoring tools.

Once again, you can find the book – all 80+ pages at this link: https://www.kentik.com/go/ebook/oreilly-guide-to-modern-networking-telemetry/.

HOW TO DEVREL: The Bechdel Test for Community

Leon Adato — Mon, 20 Jul 2026 14:00:00 +0000

DISCLAIMER: No, I don’t believe I’m the universe’s gift to DevRel. I don’t have all the answers, all the skills, or even all the Pokemon. What I have is my experiences, collected over the 11 years I’ve been doing this work. That – my experiences and observations – are what I’m sharing in this series.

For those unfamiliar with it, the “Bechdel test” is named for the American cartoonist Alison Bechdel. In one strip from her series “Dykes to Watch Out For“, a character described their 3 basic requirements to go see a movie. The movie had to have:

At least two women, who…
…talk to each other…
…about something other than a guy…

The point is that an embarrassingly small number of movies (not to mention TV shows, books, comics, etc) can clear that fairly low bar.

Equally embarrassing are the number of so-called “community forums” that fail to pass what I have termed my “Bechdel test for community”. In order to qualify as a real community, it would need to have:

at least two (non-employee) accounts with fully filled out profiles…
…talking with each other in a public area of the community forum…
…about something other than the company (or it’s products).

Meaning: A true community is a space where two people with common interests or goals talk to each other about things that matter to them. It might include the company who owns and runs the site, but it might not.

And a true community is one that not only permits, but encourages and facilitates those interactions.

Like the real Bechdel test, that bar seems low until you start surveying community sites. Most will have some form of a #random channel. Many will have other variations of #off-topic discussions. But those examples point to the flaw: often the things that permit a community to pass my test are the same things that are labeled “random” and “off-topic”, betraying the bias against having a true community.

If this hits uncomfortably close to home, don’t let that discourage you. Because improvement is within anyone’s ability.

Put Baby Product in a Corner

One of the things I find vendors fail to recognize (or know, but then quickly forget) is that their name, logo, and product are ALL. OVER. THE. PLACE. There’s need to put the brand front-and-center, nor to make the products the focus. This is because both of those things are intuitive to anyone arriving at “TechBrand.Community.com.”

Moving product-centric forums and their discussions – whether those are announcements or support discussions – into a corner effectively does the opposite of the #random channel I mentioned earlier. It makes a clear statement that “this space is for you, not for us.”

Make It a Sales-Free Zone

Closely related is a rule that I consider to be inviolable, both as a member of an online community and as someone closely involved with running them: A community space has to be completely free from sales activities.

Never in the history of ever has someone in a community forum lamented, “If only I could find a sales person to take my money!”

Like the point I made about product information, you have to recognize that community members will have no problem engaging with sales if they want it. There’s absolutely no need for sales to “check in” or “touch base” if they think a conversation is veering into sales-related territory.

I promise you that if there’s even a hit that sales folks are lurking on those channels, waiting to swoop in (either in the comments or out-of-band via email, phone calls, or whatever) your membership will shrink to zero faster than your platform’s software will be able to track.

Solicit Input, Not Free Labor

If you’re like most tech practitioners, you probably have enough work to keep you and 3 clones busy basically forever. The people in your online community are no different. If you want to demonstrate your respect for their time, not to mention your appreciation for being part of the community the last thing you should do is ask them for free labor.

That includes bug bounties, offering beta version (aka “free QA testing”), or community tech support. Even if they get points or swag.

Instead, the emphasis should be on encourating the community to share opinions, ideas, and experiences in ways that don’t look or feel like free labor.

That starts with problem-forward questions and discussions. Rather than asking how the audience uses a particular module or feature, center the conversation on “how many of you have experienced X?” (where x is a situation, issue, or problem). Or “How do you all deal with X?”

By focusing on the experience (hopefully something that happens to IT practitioners with regularity) you are making it clear your not trolling for customer reference stories.

Other conversation starters that might help:

What are some things nobody told you when you started working with (a certain technology); or in (a particular field)?
What’s a cool hack for solving X?
What issues slow you down every day that you wish you could make disappear?

…and so on.

From a company standpoint, let your community be a place where you open the door for members to participate in UX/UI sessions. These are designed to show early ideas about a new technique or technology, but in a way that allows community folks to offer input that can directly impact (and even change) the product. It’s a win-win that is still surprisingly rare in the tech world. Very few companies allow their community to get involved in a way that allows them to say “You see that option? That’s included because I told them how important it was!”

Reward Participation

That attitude – of valuing real engagement and contributions that benefit both the membership and you – should extend beyond your product to every aspect of the community. You should recognize and reward individuals any time they share their knowledge.

The place to start is recognizing and rewarding when some ASKS a question, not just when they answer them. Asking good questions is actually the hardest and most valuable element of building a vibrant and engaged community. A good question is the thing that starts a conversation, or keeps it going, or makes the answer matter in the first place.

Yes, you should also reward members who share snippets of code, scripts, dashboards, reports, and any other portable (meaning import/export-able) assets. And of course you should reward people for answering questions. But that shouldn’t be the primary focus or else you will once again fall into the trap of making the community space biased toward free labor.

Feature Request System, as a Feature

From a vendor standpoint, the most important question you can ask is “What do YOU need the product to do?”

And, as I’ve mentioned earlier when talking about UX sessions, it has to be asked in a way and at a time when the feedback you receive actually makes a difference.

For this reason, I strongly suggest making a public feature request page part of your community. This is a place where ANYONE (well, any verified community member, at least) can request a new capability.

But not only that. It should be a place where people can comment on the request to add nuance or challenge assumptions. ask clarifying questions. It’s also a place where folks can up (or down) vote those feature request.

This voting capability is more than just a popularity contest. Remember, these are requests from the public who uses your software. They are taking their time to tell you what they need (and why). When an FR gets a significant number of up-votes, it should be seriously considered by your developers. More than that, the developer staff should have to justify why those FR’s are NOT on the list, each and every development cycle.

One piece of criticism I’ve heard about this idea is “what if they say, ‘make the product free’ or ‘buy me a pony’ ?” To clarify: I’m not saying you must slavishly execute on every FR. Of course there will be folks who put in requests as a lark or just to be funny. But the vast majority will be serious.

Some – while serious – will not be possible. And that’s OK. But it’s valuable insight to know that a majority of your user base wants (for example) dark mode. If that’s something your dev team you can’t deliver, then it’s at least something your marketing team should address.

Shall We Play a Game?

Underlying all of this is a truth about online communities: the best ones make engagement and participation fun. Part of that is the content and conversations, but another part is endorphin-boosting rewards. Having a “gamified” online system that assigns points to users who answer (and ask!) questions; who upload resources; who particpate in quizes and missions; who engage in scavenger hunts and other activities… having a system that automatically tracks and tally’s reward points for those types of behaviors is important.

Equally important is having something to DO with all those points. An online shop or other system that lets community members trade in points for swag may sound foolish, but it’s the sweet icing on the cake of community interactions for many of us. It helps give us a reason to hit YOUR site rather than one of the 3 dozen other options in front of us as we sit at our screens.

Communities Are Worth It

A vibrant, engaged, participatory community is more than just a gift of goodwill to the tech space at large (although it definitely is that). It’s also an engine for change, self-awareness, and profit for the sponsoring company. It is a valuable source of customer insight – from superfans to potential employees to canaries who give advance notice when something about your product has become toxic. In one company where I worked, over $1million of revenue per year could be directly attributed to community activities.

But they require work. That includes moderation, content curation, and a careful hand when it comes to participation by the platform owner (that’s you and your colleagues) but not domination.

Hybrid Cloud – the Worst Possible Option

Leon Adato — Mon, 13 Jul 2026 14:00:00 +0000

I’m excited to share that I’m in the process of writing a book for O’Reilly Media! You can check out the Introduction and first chapter of “A Practical Guide to Modern Networking Telemetry” now. That link leads to an early release version, which means it’s been “lightly edited”. If you have comments, corrections, questions, or even kudos, let our editor know: [email protected].

I’m co-authoring it with Avi Freedman, CEO of Kentik, and it’s a labor of love for both of us. As we wrote in the “About the Authors” section, the idea to write this arose first and foremost to get down on paper all of those samples, examples, stories, and lessons we usually share in at conferences, or in talks, or when helping customers; but also to provide a resource to the readers themselves: who might need to articulate those same lessons to colleagues, managers, or the odd (very odd) person at a dinner party.

There’s one comment from the introduction that I wanted to unwind a little bit in this blog. In the section titled, “What IS the Network, Exactly”, I wrote:

Back in 2023, in his book “The Ultimate Guide to Network Observability“, Avi wrote,

“People often refer to “the network” in their organizations, but in most cases the network isn’t one entity. It’s a complex, diverse, fragmented, and loosely interconnected set of physical and virtual links and equipment, and it’s housed in a variety of places, including data centers, corporate wide area networks (WANs), private and public clouds, the internet, container environments, and even inside hosts. Organizations own and control some of those resources but simply pay to use others.”

Since that time, things remain largely the same. Sure, the cloud has gotten more cloudy (meaning opaque to users) even as it has become more ubiquitous within organizations large and small. And at the same time corporate LAN and WAN environments have not, by and large, become less complex (or sprawling, or expensive). In fact, the opposite has been true. Even as company’s investment in cloud has increased, so too has their investment in on-premises networking gone up.

Why is that? Because the more we – businesses and IT professionals alike – use the cloud, the better we understand what the cloud should be used for; and conversely, where cloud should absolutely not be used, whether due to considerations of cost, complexity, or security. And in those cases, we have brought systems, applications, data, and solutions back into our data centers, and found new and creative ways to connect and combine it with the cloud infrastructure.

And that – the cursed franken-network of modern IT: infrastructure that spans both cloud and on-prem environments, offering all the complexity, cost, and issues of both, while offering very little of the benefits.

The sheer horrific nature of hybrid networking might not be intuitively obvious to folks reading this because we live and breathe in a technical context every day. So let me draw a parallel that might make it clearer: hybrid cars.

Blending a traditional gas-powered combustion engine with a modern fully-electric motor sounds like a great idea on paper, but in practice it’s a nightmare. The design burdens the user with all the quirks and complexities of both systems; it also creates a completely new set of quirks and complexities on top of those; and offers only modest benefits – which are usually completely outweighed by the negative aspects of smushing the two technologies into one form factor.

And that’s what hybrid networks are: the combined costs and complexities of on-premises and cloud networks, with a whole new layer of the same incurred by trying to make them play nicely together.

The big difference between hybrid cloud and hybrid cars is, of course, that we IT practitioners don’t have much of a choice. As much as we wish – hell, as much as we TRIED over the last decade to push environments to fully-cloud or fully-on prem, hybrid is the reality we’re going to live in for the foreseeable future.

And that means we – IT practitioners in general and folks who care about (and are responsible for) monitoring and observability in particular – have to improve the ways we ensure performance and reliability of those environments.

And here, approximately 700 words into this blog, I’ve finally gotten to the problem I hope “The Practical Guide” will help fix:

the folks who are often responsible for building the cloud-based side of those environments often know very little about networking
and meanwhile, the teams who are often responsible for building the on-prem side of those environments often know very little about modern observability.

The book offers a clear and complete explanation of everything from the types of devices that comprise a “network”; to the types of telemetry those devices emit; and continues from there to explain how to collect, transform, transmit, and use that telemetry.As I mentioned at the start of this blog, chapter 1 is available in its early release version now. Go ahead and give it a look, and let me know what you think – either in the comments below, or by emailing my editor Gary at [email protected].

Time Data Series: Written In Our Stars

Leon Adato — Wed, 08 Jul 2026 13:49:37 +0000

Once upon a time (i.e. way back in October 2024) I wrapped up a blog in this series with:

Believe it or not, there’s still more to cover. In the coming weeks whenever I get to it, I’d like to cover ways to leverage the built-in astronomy functions for time calculations. Believe it or not, it can really matter in terms of having truly accurate times for things like the start and end of Shabbat and holidays, along with other observances.

And here we are in 2025, when it’s finally time for me to fulfill that promise, and talk about how the KosherJava library (and by extension, the PHP Zmanim port I’ll be using in this blog) can help you calculate and display what amount to straight astronomy in your application or web page.

Once again I need to begin by expressing my deep gratitude to Zachary Weixelbaum and Eliyahu Hershfeld, who both continue to answer so many of my persistent (and often basic) questions with patience, care, and good humor.

The House of the Rising Sun

The core aspect of what I’m showing is fairly simple: For a specific location (latitude/longitude, plus timezone) and date, find the time of day that the sun will be at a given elevation point.

Or to be even more specific: At what time will the sun be at 20 degrees above the horizon on June 10, 2025 in Cleveland? So let’s break down the key elements of PHP Zmanim we’ll be focusing on today.

These two lines load the parts of the PHP library we’ll be using.

use PhpZmanim\Geo\GeoLocation;
use PhpZmanim\Calendar\ComplexZmanimCalendar;

Next, we set up a location object (using variables for location name, latitude, longitude, elevation, and time zone); and a calendar object (using the location object we just instantiated along with the year, month, and day):

$location = new GeoLocation($locname, $lat, $long, $elev, $tz);
$zmanim = new ComplexZmanimCalendar($location, $getyear, $getmonth, $getday);

Finally, the code that calculates and displays the results:

$sunrise = $zmanim->getsunrise();
$twenty = $zmanim->getSunriseOffsetByDegrees(70);
print("sunrise: $sunrise\n20 degrees: $twenty\n");

If playing along at home, I should explain that I grabbed both the regular sunrise simply as a reference to the time at which the sun is 20 degrees up in the sky.

I also need to explain that yes, I know that 70 is not 20, so how does THAT work out? The getSunriseOffsetByDegrees function “thinks” of the sun as being at 0 degrees when it’s at its zenith, and 90 degrees at the horizon line. So for the purposes of this function going up (higher in the sky) causes the degree value to go down.

The full code looks like this:

use PhpZmanim\Zmanim;
use Carbon\Carbon;
use PhpZmanim\Calendar\ComplexZmanimCalendar;
use PhpZmanim\Geo\GeoLocation;

# Set variables:
#41.4939407, -81.516709;
$locname = "Beit Knesset Chochmat Shlomo, Beachwood, OH";
$lat = 41.4939407;
$long = -81.516709;
$elev = 0;
$tz = 'America/New_York';
$getyear = "2025";
$getmonth = "06";
$getday = "10";

$location = new GeoLocation($locname, $lat, $long, $elev, $tz);
$zmanim = new ComplexZmanimCalendar($location, $getyear, $getmonth, $getday);
$sunrise = $zmanim->getsunrise();
$twenty = $zmanim->getSunriseOffsetByDegrees(70);
print("sunrise: $sunrise\n20 degrees: $twenty\n");
?>

When the Night Has Come, and the Land is Dark…

I’m sure there are useful reasons to know the time when the sun is at a certain point in the sky during the day, but from a religious perspective, there are far more reasons to want to know when the sun is below the horizon line. I say “religious” because I can think of at least two that care about the time of the sun’s position before sunrise: Judaism and Islam.

Both have a concept of dawn that indicates the very first visible lightening of the sky in the east. This time is known as fajr in Islam and alot hashachar in Judaism.

Because we’re talking about a matter of perception, it can be ferociously difficult to nail down, a matter not made any easier by the constant shifting of the sun over the year, and the differences due to latitude.

However – and bypassing a huge swath of religious debate – let’s accept the opinion that the first light is visible by perceptive humans on a clear day when the sun is 18

degrees below the horizon. How can we calculate that?

Honestly, it’s not hard now that you know the basic structure. In fact, we just have to change one thing:

$dawn = $zmanim->getSunriseOffsetByDegrees(90+18);

Yes, I could have just typed 108 but I wanted to make it explicit that we’re calculating 18 degrees before sunrise (90 degrees from the zenith).

Get Me to the Church Synagogue on Time…

To offer a more involved – not to mention specific – example of how this can be used, I’d like to explain one method of finding the time for “Tzeit haKochavim” (“nightfall”, which is more poetically described as the time when it’s dark enough to see 3 large stars in the sky). For this particular method, the following series of calculations are used:

Select the date of the equinox (spring or fall)
Get the time for sunset
Get the time when the sun is 3.86 degrees below the horizon (after sunset)
Get the difference in minutes between those 2 times
Select the current date
Calculate “seasonal minutes” – an even division of daylight 6.1) Subtract sunset from sunrise to get the total amount of daylight 6.2) Divide that time into 12 even “seasonal hours” 6.3) Divide 1 seasonal hour into 60 even “seasonal minutes”
multiply the clock minutes you got from the equinox day calculations and multiply them by seasonal minutes.
And finally, add THAT number of minutes to sunset, to get the time of “nightfall” / “Tzeit haKochavim”

Confused? Yep, me too. It took me a while to wrap my head around this. But here’s what it looks like in code:

First, let’s set our location and today’s date:

$locname = "Beit Knesset Chochmat Shlomo, Beachwood, OH";
$lat = 41.4939407;
$long = -81.516709;
$elev = 0;
$tz = 'America/New_York';
$getdate = date('Y-m-d');
$getyear = date('Y', strtotime($getdate));
$getmonth = date('m', strtotime($getdate));
$getday = date('d', strtotime($getdate));

Next, we’re going to create a location object, and use that location object to create a calendar object set to an equinox day (in this case March 20).

$location = new GeoLocation($locname, $lat, $long, $elev, $tz); 
$eqzmanim = new ComplexZmanimCalendar($location, $getyear, '3', '20');

Get sunset for that day, along with the time the sun is 3.86 degrees below the horizon

$eqsunset = $eqzmanim->getsunset();
$eqtzeit = $eqzmanim->getSunsetOffsetByDegrees(90+3.86);

Get the number of minutes between sunset and 3.86 degrees after sunset.

$tzeitmin = number_format((float)((strtotime($eqtzeit)-strtotime($eqsunset))), 2, '.', '')/60;

OK, now it’s time to get some calculations for TODAY’S date. We’ll create a new calendar object and grab sunrise and sunset:

$todayzmanim = new ComplexZmanimCalendar($location, $getyear, $getmonth, $getday);
$sunrise = $todayzmanim->getsunrise();
$sunset = $todayzmanim->getsunset();

Now we’ll divide the time between sunrise and sunset into 12 equal parts (a “sha’ah” in Hebrew), and then divide one sha’ah into 60 even parts. That will effectively give us the ratio of seasonal minutes to clock minutes

$stdshaamin = number_format((float)((strtotime($sunset)-strtotime($sunrise))/12)/60, 2, '.', '');
$stdzmaniot = $stdshaamin/60;

Now we multiply the number of minutes from our equinox day times the seasonal minute ratio.

$tzeitadjustment = $tzeitmin * $stdzmaniot;

And finally, we add that adjustment to today’s sundown (also note we have to multiply the adjustment * 60 to make the clock math work out).

$todaytzeit = date('Y-m-d G:i:s', strtotime($sunset) + $tzeitadjustment*60);

And let’s output it to see what we get:

print("sundown: $sunset\ntzeit: $todaytzeit\nadjustment: $tzeitadjustment\n");

For June 10, 2025, the output would look like this:

sundown: 2025-06-10 20:59:34
tzeit: 2025-06-10 21:19:59
adjustment: 20.4228

Good Night, My Someone

Believe it or not, we’re reached the end of my “Time Data Series”. I will, of course, keep adding things as I learn them myself. But – unless you have something specific you’d like to see (which, by all means, please tell me about in the comments), I’m going to turn out the lights, take a moment to admire the stars both for their beauty and the breathtaking wonder at the complexity of their movements, and head to bed.

Thank you for coming along on this ride with me.

HOW TO DEVREL: Fuel for Your Creativity

Leon Adato — Fri, 04 Jul 2025 01:54:58 +0000

In my last post I talked about re-factoring your content from one form into another (blog to video, video to conference talk, conference talk to podcast, etc.). Which is good as far as that idea goes, but there is so much more you can do.

In fact, let’s do a thought exercise:

The Idea Engine Challenge

Challenge #1: Put 1 minute on the timer, take your hands off your keyboard / phone (meaning: NO INTERNET!) and then:

name 10 (or more) sports teams

How many did you get? How many more could you have come up with if you had more time? How many could you have gotten if you could have used the internet?

Challenge #2: Put 3 minutes on the timer, take your hands off the internet, and then:

List 25 (or more) songs with the word “love” in the title

Good work! Once again, take a moment to reflect: How many more could you have done with more time, with the internet, or if you could have worked with a partner.

Challenge #3: Put 5 minutes on the timer, once again no internet, and then:

List 50 (or more) recipes – just the name/title, not the ingredients

And for the last time, take time to think about how many more you could have done with help (internet, friends, more time).

The Real Challenge

Now give yourself a luxurious 10 minutes on the clock. You’re allowed to use the internet, or phone a friend, or whatever else you want. And see if you can:

List 100 content ideas

OK, stop (whether it’s been 10 minutes, or 1 minute, or no minutes at all). If you’re like most folks, that challenge felt distinctly different, and not in a good way. If so, the first thing to understand is that it’s totally normal. Coming up with content ideas isn’t the same as coming up with city names or sports teams or song titles.

The second thing to consider: Why? What was so different about coming up with content ideas versus songs or recipes? Likely, one of the primary reasons is that, whether consciously or not, you had a system in place for the first 3. You probably started listing:

Sports teams from places where I’ve lived
Pop songs from high school (or some other significant period of your life)
Recipes I know how to make
…etc…

BUT… your brain, if it’s anything like mine, gave no such immediate framing when it came to the challenge for content. Again, if you’re like me, reading that question felt like opening a door onto a vast barren, featureless wasteland where there was very little to latch onto, to connect the dots with, to even build a series of ideas around.

Framework is the Key

That is the brilliant observation behind Melanie Deziel’s book, “The Content Fuel Framework“. In the book, she outlines a few central issues with content creation, and then offers a single conceptual solution which is simple, repeatable, and effective.

I’m going to stop here and ask you to at least click the link and consider buying the book. I believe it’s that good. In fact, if you plan to buy the ebook or audiobook version, then your time will be better spent reading THAT instead of THIS. But if you’re still on the fence, by all means keep reading so that I hopefully convince you by the end of this post.

Format Follows Focus

“I NEED AN 800-WORD BLOG POST ON ______!”

These words, uttered by far too many of my managers over the years, are as frustratingly vague as they are common. I’ve taken to muttering (usually under my breath and after ensuring my zoom call is on mute) “Would you like fries with that?” When I’m feeling slightly less snarky I try to get some details. “OK, so you want 800 words on Kubernetes and IOT. Anything specific ABOUT kubernetes and IOT?”

“Nope!” Chirps this (only slightly fictionalized) manager, “The SEO is trending on those two terms, so I just need you to, you know…” (and then they waggle their fingers at the computer screen as if casting a spell.)

Ignoring the whole “SEO-term-driven content management strategy of doom” for a moment, a deeper issue is the belief that CONTENT == BLOG. Content can, obviously, take many forms. But equally problematic is when we are asked to “make me a video on ______”, or “Can you whip up a podcast episode about _______”.

The issue, Ms. Deziel points out, isn’t the fact that they are asking for content, it’s that they (and we) are starting with the format, and working backward to a topic. Worst of all is that a distinct framing, or point of view, isn’t even part of the consideration.

First and foremost, “The Content Fuel Framework” advises, we need to start with the FOCUS of the content piece.

What do I (and more importantly, Ms. Deziel) mean by “focus”? It’s not the topic. Your topic can be kubernetes, or IOT, or mainframes, or kubernetes mainframes running on IOT, or whatever.

Your Focus is “…the approach we take to sharing [our message].” If you feel like that description is unhelpfully abstract, let me offer some examples. Your focus might be:

Basics
Details
People
History
Opinion

The first two are the easiest types of focus to understand. For “Basics”, just think of a content piece titled “Getting started with _____”. Meanwhile “Details” is the grown-up cousin of Basics. Imagine a title like “Everything you need to know about _____” or “A S]Step by Step Guide to _____.”

But the other focus examples above are less obvious. “People” might include a title like “What led John Smith to invent the mainframe?” or “A profile of the top 3 movers and shakers in the world of kubernetes today”.

In her book, the author shares 10 focus areas, with detailed descriptions of how they work. My goal today is to give you a sense of how a “focus” works, but not to outline them all.

Once a focus is chosen, then (and, I’d argue, ONLY then) can you consider the format. Some of the formats are obvious – writing, audio, and video. Some might not have leapt to your mind but make sense once you see them – Infographic, timeline, Quiz. Others require explanation to fully understand, like “Tool” and “Map”.

Again, my goal here isn’t to list or describe them all, but to indicate that as content creators, our habit of always reaching for the same old formats doesn’t serve us well. Expanding our range of options allows us to find new angles to tell familiar stories; and to find formats that better suit those stories rather than shoehorning everything into a blog or a stare-at-the-camera-and-talk style video.

Greater Than The Sum of It’s Parts

“The Content Fuel Framework” offers 10 focuses (focii?), and 10 formats. These are somewhat interchangeable, meaning you could start with a “People” focus and create a blog, AND a video, AND a podcast, AND an image gallery, AND…

I’m no Alfred Einstein when it comes to math, but I’m pretty sure that 10 x 10 gives us, theoretically, a single topic (Kubernetes and IOT) with 100 content pieces.

Realistically, you’re not going to do that. Some focuses simply lend themselves to certain formats better than others. More importantly some core topics will more readily lend themselves to certain focuses than others.

But having explained The Content Fuel Framework, I’d now ask you to scroll back and consider the challenge I set before you at the start of this blog: 100 content ideas in 10 minutes. As you think about how far you could get before the buzzer sounds, it’s my sincere hope that you feel better, if not equal to the challenge.

Through Structure, Freedom

It turns out that having a framework – a set of rules, guidelines, even limitations – can create clarity by eliminating ambiguity.

I’m not implying that this is the only way, or even the One True Way to create content. “The Content Fuel Framework” is just one way to break out of a rut and find a new way to approach what is, for most of us, a significant part of the day to day work of DevRel. There are, of course, many others ways (feel free to share them in the comments).

More importantly, your newfound confidence (however you got it) in generating ideas, angles, and implementations for content will translate to an improved ability to collaborate with the teams at work: First, you hopefully now have a structure you can use to teach them that there’s more to DevRel than writing a blog; and from there you have ways to help them tell the best possible stories. Stories that reach, help, and resonate with the greatest number of people.

HOW TO DEVREL: What I Learned From "Spider-Man, The Musical"

Leon Adato — Fri, 04 Jul 2025 01:53:33 +0000

You wrote a blog. You’re happy with it. Proud of it, even. You like the way you captured your voice and style in the written form. You also like what the blog says – what it teaches the reader. It’s a good little blog. In your imagination, you give it an affectionate little pat on the head. You click “Post” and make sure it shows up on the public-facing site. Maybe you even give it one more little read-through.

OR… you deliver a conference talk. The slides keep people’s attention, your delivery and timing is spot-on. You get laughs at just the right moments, and there are plenty of questions for you at the end. As the applause ends and people head for the next session, you take a deep breath, unplug your equipment, and close the slide deck.

OR… you created a video. Again, you’re happy with it – maybe even a little proud – for all the same reasons as I mentioned previously. You give it one more watch, then click “Publish”.

Time to move on, right?

Then you get a message. It might be from someone in the company, it might be a contact on social medial. Heck, it might even be your Mom. But the message says “You should turn this into a blog/video/conference talk!” Meaning, take it from the form it’s in now, and turn it into one of the other forms (or any one of a few different options that I’ll get to shortly).

Should you though? I mean, you just SAID everything you have to say. Why say it again? Why water down the power, effectiveness, and presence of the initial form by creating a derivative work?

An Intertwining Web of Tales

This is the dilemma we (creative folks in general, and technical content creators in particular) face when we consider the idea of “re-use”. And I’ll tell you why you should do it. Or at least strongly consider doing it: Because “Spider-Man, The Musical” exists.

Here’s tl;dr of what I mean: the story of “Spider-Man” exists in many many forms. But each format works in it’s own unique way that is complimentary to, but separate from, the others.

Spider-Man, of course, began his life on the printed page. But even there, a single comic book series wasn’t enough. Since it’s inception in 1962 there have been more than 60 different serialized Spider-Man story-lines, with multiple series running simultaneously for much of that time.

Then came the Saturday morning cartoon. It wasn’t better OR worse than the comic. It was DIFFERENT. It was able to do things the pulp fiction version couldn’t. And of course, there were things the paper version could do that could never make the translation to animation. Likewise the 1977 live-action tv series was able to present things that were missing in both the printed and animated stories.

And so it goes – from print to animated tv to live tv to live movies to animated moves and around and around, like a spider spinning it’s web – each form able to highlight and even elevate some aspects, but unable to depict others.

I would continue this entire analogy to the musical, but in reality “Spider-Man: Turn Off the Dark” struggled in just about every way. So I’m going to pivot to a different story to continue my point

The Circle of Creative Content

The Lion King was an award winning movie, a magical experience for those who were a certain age in 1994 when it came out. When the musical was announced in 1997, I was skeptical. How could a live action portrayal of Simba’s journey ever capture the sweeping majesty and raw power of the movie? And of course, my skepticism was utterly unfounded. The musical took such a different, and equally majestic form that it captured imaginations in a way the movie could not.

Pivoting further away from wall-crawling web-heads, there are plenty of other cases where an original creative work was transformed – either in it’s entirety or in it’s essence – to a different form that could arguably be said was as strong as, if not stronger, than the original. Just one such example is when Eric Clapton transformed the wailing rock song “Laylla” into a jazzy and almost haunting acoustic version for MTV’s “Unplugged”.

Revisiting the Question

Coming back to the initial dilemma: Should you take your original content and transform it to a different format? My answer should be obvious: of course. Taking a video and blog-ifying; taking a video and blogging it; taking a conference talk and recording it; each of these modulations from one format to another will allow you to elevate, highlight, and showcase aspects of the core narrative in ways that the original delivery could not.

Moreover, the work of transforming it will force you to consider things from an entirely different point of view, and this might shine a light on elements or ideas that you completely overlooked during your first pass – either because the original format couldn’t support it or because the very act of translation requires re-evaluation.

Far from being watered down, you may discover that your story – whatever form it’s in – becomes stronger, more cohesive, more well-defined as a result.

HOW TO DEVREL: The Texas Sharpshooter

Leon Adato — Thu, 03 Jul 2025 19:49:22 +0000

DISCLAIMER: I do not believe that I’m the universe’s gift to DevRel. I don’t have all the answers, all the experiences, all the skills, or even all the Pokemon. What I do have is my experiences, collected over 11 years of doing this work. That’s what I’m sharing in this series.

Work at a software vendor (or really any group that creates features for a set of consumers – whether those consumers are internal users or external customers) for any length of time and inevitably you experience a moment when a new feature, module, or capability is released to the public, and the response is the human equivalent of this:

It becomes clear that the intended audience has no idea what the new feature is, why it’s there, or – most critically – what problem or use-case it addresses.

This is not (or at least not always) because the feature is badly designed, ill conceived, or poorly executed. Sometimes, the response from users is because the problem hasn’t been socialized well enough. Nobody understands the nature of the disease, and therefore they can’t understand the importance of the cure.

As a DevRel, almost as important as our work helping folks understand HOW (to install, to use, to extend) is explaining WHY: Why this capability is necessary? Why we decided to create this instead of 6 other things on our backlog? Why this feature provides an optimal way to solve a particular problem.

But I’ll be honest: Explaining HOW is relatively easy. Explaining WHY can be a lot of work.

One of the techniques I use that makes explaining WHY easier is a variation of the Texas Sharpshooter story. While this is normally categorized as a logical fallacy (and, when used in data analysis or debate, it is), for Developer Relations Advocates it is actually a fairly useful technique.

The Man, the Myth, the Legend

I’ll start with the colorful story behind the evocative name:

Once there was a prince who was also an accomplished marksman, who was traveling through the land. Along the road he stopped at a village to rest his mount. He sat down under a tree—and right there in front of him was the side of a big barn. He couldn’t believe his eyes. That whole side of the barn was covered with little circles, drawn with chalk, and in the center of each circle was a bullet hole! He jumped up in amazement. Who is this sharpshooter around here who never misses? Maybe they could have a little contest while he was here.

So he knocked on the door of the farmhouse and asked if they knew who the marksman was. Yes, they did. They brought him a barefoot boy of about twelve. ‘Here he is, Your Excellency, our son the sharpshooter!’ Of course, the prince was surprised. He asked the boy, ‘Tell me, young fellow, how do you manage to hit the bullseye all the time? Here, look at all these medals. I’ve just finished five years of training and even I couldn’t hit the target every single time.’

‘Oh,’ said the boy, ‘there’s nothing to it. I just like to shoot. So I shoot at the side of the barn and then I take a piece of chalk and draw a circle around the hole.’

Context Matters

Shooting the bullet and then drawing the target later makes for a bad-faith argument, but it turns out it’s perfect for content-planning.

When you work “inside”, meaning you are working with the developers of a product, you are actually existing weeks or even months in the future. What I mean is, your day to day thoughts and internal discussions are about a future version: how it will work, the problems it solves, etc. Whereas the users of the product only know about the features and capabilities they have today.

It’s often a challenge to remember that your context – the problem you’ve been working on for weeks, the one that will be solved with this next release – is completely unknown and unexpected by the majority of users. They haven’t been pondering the complexities of the use case. THEY haven’t been investigating ways to remove friction, toil, or error.

Your v-dot-now is their v-dot-next.

The mistake many organizations (and the DevRel’s right along with them) make is releasing the new version with the expectation that everything – the problems, use cases, and purpose of the new version – will be intuitively obvious because, duh, we’ve been talking about it FOREVER, right?

That’s how you get the German shepherd look when you do the big reveal.

YOU know what the feature is going to do, what problems it will solve, what improvements it will bring. BUT… you also know it won’t be ready for a little while. Talking about it now, when features could slip, deadlines might be missed, priorities re-arranged… heck, when entire teams could be re-assigned… promoting the solution now would be asking for problems when things don’t go as planned.

But you also can’t talk about nothing. And that’s where the Texas Sharpshooter technique comes in.

The new feature is the metaphorical bullet. It’s already been shot, in the sense that development work is already happening. You know where it hit, so to speak. Despite the fact that nobody but your internal team can see it, the landing point is there, on the wall. You’re job right now is to draw a circle around that spot by socializing the problem. You draw attention to the problem, the point where the bullet WILL appear, by drawing a big old bull’s eye on that spot.

In blogs, opinion pieces, panel conversations, videos, or whatever, your job as a DevRel is to talk up the challenge, cost, and toil of the issue – what it is, why it’s a problem, how much damage it does, and the current (imperfect, complex, kludge-y) solutions that exist.

Period. That’s it. No parting of heavenly clouds, no angelic choir. Just you and the audience bonding over your shared experiences. You give a knowing look that says “Life in tech, amirite?”

By doing this, you establish the context for the eventual unveiling.

TAMO (then a miracle occurred)

The best part of this technique is that you can have multiple threads like this going at the same time. Because let’s face it, we’re in tech and we looooove to complain. And show off. And complain about showing off. And show of how we compla…. never mind. You get it.

My point is that the companies where we typical work as DevRel Advocates usually have multiple new features, modules, and improvements in the pipeline at any given time. Maybe only a couple will hit GA soon, but we can get ahead of the curve by socializing those problems and fostering the online discussions, debates, counterpoints, etc long before the release date.

More than just being a clever marketing ploy, the sharpshooter concept also allows the business to understand how customers and the community will ultimately react to the new release. By gauging the audience’s overall understanding of the technologies and techniques, the organization can have improved documentation documentation or educational resources ready to go. By listening to the specifics of the dialogue about the problem itself, adjustments (or even UX/UI sessions) can be added to the timeline.

When used thoughtfully, this technique allows for a better overall product as well as a better overall release.

HOW TO DEVREL: It Might Not Be New, but It's Yours

Leon Adato — Thu, 03 Jul 2025 19:47:08 +0000

Lately, I’ve had several conversations about “how to DevRel”. Some of that is absolutely related to my current job search. But it’s also been a topic that comes regularly even when I’m settled into a company: When meeting folks at a conference who want to understand what my title (which has fluctuated between “DevRel Advocate”, “Technical Evangelist”, and even “Head Geek”) means; Or discussing specific techniques with other DevRels on our super secret slack channel; Or justifying my existence internally because, even though the role has existed since 1980 half the company still doesn’t know what the frak it is, nor what it involves.

Although that last part is more due to the fact that “Developer Relations” is one of those five-jobs-in-a-trench-coat types of roles, so it’s understandable.

In any case, I’ve been mulling over not just what DevRel IS, but also the ways I approach the work of DevRel, from writing to video to community to speaking to demos and onward. And I thought I’d blog about it for a bit.

DISCLAIMER: I do not believe, for a moment, that I am the universe’s gift to DevRel. I don’t have all the answers, all the experiences, all the skills, or even all the Pokemon. What I do have is MY experiences, and while they aren’t the last word in DevRel, they are uniquely MY word about DevRel, collected over 11 years of doing this work. That’s what I plan to share.

We’ll call this my “How To DevRel” series, and see where it takes us. Today I want to address a common concern among content creators of all stripes, which is the fear that the thing you’re creating has already been created in some form by someone else, and who am I to take another swing at that topic now?

Nothing New Under the Sun

I’ll start by selectively cut-quoting Marianne Williamson (you can read the full quote here)

We ask ourselves, ‘Who am I to be brilliant, gorgeous, talented, fabulous?’ Actually, who are you not to be? […] Your playing small does not serve the world. There is nothing enlightened about shrinking so that other people won’t feel insecure around you. We are all meant to shine, as children do.

You’re afraid you’ll say something that has been said before? That you’ll demonstrate a tool or technique that’s been covered a dozen (or a hundred, or a thousand) times by the original inventor, and by someone from the inventor’s company, and by 6 different IT luminaries, (and so on, all the way down to the inventor’s dog)?

Let me put that worry into a different context to see if it still holds water:

Do you remember when you heard the song “White Christmas” that one time, and of course you never had to listen to it again, because you heard it already? How about the fact that Bing Crosby sang it just once, because he didn’t want to “be derivative”? And of course, nobody has ever sung “White Christmas” ever since, because of course Bing sang it and so it’s been sung and that’s all there is to it.

Of course not. The premise that we experience art once, that the first rendition is the only authentic one, is ridiculous.

Not only do we listen to the same recording over and over, many folks go out of their way to see the same performer sing the exact same song (or set of songs) multiple times. We go to hear cover bands play the exact same set but of course, with their own personal twist on things. We go to multiple renditions of a play or musical when they show up in our town.

In short, a story (whether it’s written or spoken or sung or danced or painted) becomes more powerful, not less, when we hear it through the lens of a different person’s experience and style.

Imitation is…

Yeah yeah, we all know that quote. And also the one Picasso said about stealing.

But here’s something important to understand: Imitation is when you do something that someone else has already done in the same way they did, without adding anything of your own, with the expressed intent of doing it that way BECAUSE the other person did it that way.

For creative folks, pure imitation is actually really REALLY difficult. It’s nearly impossible NOT to add our own spin on something. Even comedians who do impressions of other people understand that the real humor lies in “doing” the other person with just enough of a twist that it’s not just different for the sake of being different, it’s different in a small, subtle, and unexpected way. That’s what makes it funny.

On the other hand, even creations that were themselves inspired by other creations bring something new. “It Ain’t Me Babe” by Bob Dylan was inspired by The Beatles “She Loves You“. Sheryl Crow’s “All I Wanna Do” was inspired by Wyn Cooper’s poem “Fun“. Claude Monet’s “Déjeuner sur L’Herbe” was inspired by Edouard Manet’s painting of the same name. To go a little further afield, the Broadway musical “Sunday in the Park with George” was, of course, inspired by the Seurat painting “Un dimanche après-midi à l’Île de la Grande Jatte” (A Sunday Afternoon on the Island of La Grande Jatte) .

It’s OK to find inspiration and insight in the work of others, to build your story out of something you saw, read, or heard. Of course give credit where it’s due, but also trust that your creation is going to have it’s own new elements that make it fresh, valuable, and uniquely yours.

Saying it With Your Voice Adds Power

When you write (or make a video, or talk at a conference) about a new technique, technology, or experience – one that’s not NEW-new, but rather new-to-you – you open the door not only to folks who, like you, haven’t known about it but also you allow people who were already familiar with it the chance to see it through your eyes. Your excitement, curiosity, and enthusiasm cast the well-known into a new light. You also create the opportunity to review and re-think that topic, to reconsider it’s value or application today.

I believe in this idea so strongly, I took one of my conference talks – “Looking Back at a Lifetime of Poor Technical Choices“, and made it open source. I realized that even using the exact same format and structure, everyone’s journey and choices would come out so differently, and resonate so powerfully, that you could do 5 of the same talk in the row and they would all be interesting, compelling, entertaining, and informative. You can find it (and download it, and make it your own) here on my github repo.

So… If you find yourself asking, “Who am I to give this demo? To write about this topic? To host this podcast? To be on this panel?” then, to paraphrase Ms. Williamson, “…Who are you NOT to?”

She continues, powerfully,

“As we let our own light shine, we unconsciously give other people permission to do the same. As we are liberated from our own fear, our presence automatically liberates others.”

OTel Me Why

Leon Adato — Mon, 07 Apr 2025 04:00:00 +0000

LIMITED TIME OFFER! I’m ready for my next adventure as a DevRel advocate / Technical Evangelist / IT Talespinner. If that sounds like something you need, drop me a line in email or on LinkedIn

My blog on pricing from the other day caught the attention of the folks over at MetricFire, and we struck up a conversation about some of the ideas, ideals, and challenges swirling around monitoring, observability, and its place in the broader IT landscape.

At one point, JJ, the lead engineer, asked, “You blogged about gearing up to get a certification in Open Telemetry. What is it about OTel that has you so excited?”

I gave a quick answer, but JJ’s question got me thinking, and I wanted to put some of those ideas down here.

OTel is the best thing since…

Let me start by answering JJ’s question directly: I find Open Telemetry exciting because it’s the biggest change in the way monitoring and observability are done since Traces (which came out around 2000, but wasn’t widely used until 2010-ish).

And Traces were the biggest change since… ever. Let me explain.

See this picture? This was what it was like to use monitoring to understand your environment back when I started almost 30 years ago. What we wanted was to know what was happening in that boat. But that was never an option.

We could scrape metrics together from network and OS commands, we could build some scripts and db queries that gave me a little bit more insight. We could collect and (with a lot of work) aggregate log messages together to spot trends across multiple systems. All of that would give us an idea of how the infrastructure was running, and infer the things that might be happening topside. But we never really knew.

Tracing changed all that. All of a sudden we could get hard data (and get it in real time) about what users were doing, and what was happening in the application when they did it.

It was a complete sea change (pun intended) for how we worked and what we monitored. Even so, tracing didn’t remove the need for metrics and logs. And famous (or infamous) “three pillars” of observability.

Recently I started working through the book “Learning OpenTelemetry” and one of the comments that struck me was that these aren’t “three pillars” in the sense that they don’t combine to hold up a unified whole. Authors Ted Young and Austin Parker re-framed the combination of Metrics, Logs, and Traces as “The three browser tabs of observability” because many tools put the effort back on the user to to flip between screens and put it all together by sight.

On the other hand, OTel outputs is able to present all 3 streams of data as a single “braid”

It should be noted that despite OTel’s ability to combine and correlate this information, the authors of the book point out later that many tools still lack the ability to present it that way.

Despite it being a work in progress (but what, in the world of IT, isn’t?) I still feel that OTel has already proven its potential to change the face of monitoring and observability.

OTel is the Esperanto of Monitoring

Almost every vendor will jump at the chance to get you to send all your data to them. They insist that theirs is the One True Observability Tool.

In fact, let’s get this out in the open: There simply isn’t a singular “best” monitoring tool out there any more than there’s one singular “best” programming language, or car model, or pizza style.* There isn’t a single tool which will cover 100% of your needs in every single use case.

And for the larger tools, even the use cases that aren’t part of their absolute sweet spot are going to cost you (in terms of hours or dollars) to get right.

So you’re going to have multiple tools. It goes without saying (or at least it should) that you’re not going to ship a full copy of all your data to multiple vendors. Therefore a big part of your work as a monitoring engineer (or team of engineers) is to map your telemetry to the use cases they support, and thus to the tools you need to employ in those use cases.

That’s not actually a hard problem. Sure, it’s complex, but once you have the mapping, making it happen is relatively easy. But, as I like to say, it’s not the cost to buy the puppy that is the problem, it’s the cost to keep feeding it.

Because the tools you have today are going to change down the road. That’s when things get CRAZY hard. You have to hope things are documented well enough to understand all those telemetry-to-use-case mappings.

(Narrator: they will not, in fact, have it documented well enough)

Then you have to also hope your instrumentation is documented and understood well enough to know how to de-couple tool x and instrument tool y such that you maintain the same capabilities.

(Narrator: this is not how it will go down.)

But OTel solves both the “buying the puppy” and the “feeding the puppy” problem. My friend Matt Macdonald-Wallace (Solutions Architect at Grafana) put it like this:

OTEL does solve a lot of the problems around ‘Oh great! now we’re trapped with vendor x and it’s going to cost us millions to refactor all this code’ as opposed to ‘Oh, we’re switching vendors? Cool, let me just update my endpoint…’

Not only that, but OTels ability to create pipelines (for those who are not up to speed on that concept, it’s the ability to identify, filter, sample, and transform a stream of data before sending it to a specific destination) means you can send the same data stream to multiple locations selectively. Meaning your security team can get their raw unfiltered syslog while it’s still on-premises. Some of the data – traces, logs, and/or metrics – can go to one or more vendors.

Which is why I say: OTel is the esperanto of observability.

OTel’s Secret Sauce isn’t OTLP

…it’s standardization.

Before I explain why the real benefit to Otel is not OTLP, I should take a second to explain what OTLP is:

If you look up “What is Open Telemetry Line Protocol?” you’ll probably find some variation of “…a set of standards, rules, and/or conventions that specify how OTel elements send data from the thing that created it to a destination”. This is technically true, but also not very helpful.

Functionally, OTLP is the magic box that takes metrics, logs, or traces and sends them where they need to go. It’s not as low level as, say, TCP, but in terms of how it changes a monitoring engineer’s day, it may as well be. We don’t use OTLP so much as we indicate it should be used.

Just to be clear, OTLP is amazingly cool and important. It’s just not (in my opinion) AS important as some other aspects.

No, there are (at least) two things that, in my opinion, make OTel such an evolutionary shift in monitoring:

Collectors

First, it standardizes the model of having a 3-tier, collector (not agent) in the middle, architecture. For us old-timers in the monitoring space, the idea of a collector is nothing new. In the bygone era of everything-on-prem, you couldn’t get away with a thousand (or even a hundred) agents all talking to some remote destination. The shift to cloud architecture changed all that, but it’s still not the best idea.

Having a single (or small number) of load-balanced systems that take all the data from multiple targets – with the added benefit of being able to then process that data, filtering, sampling, combining, etc – before sending it forward is not just A Good Idea™, it can have a direct impact on your bottom line by only sending the data you WANT (and in the form you want it) out the egress port that racks up such a big part of your monthly bill.

Semantics

Look, I’ll be the first to tell you that I’m not the world’s best developer. So the issue of semantic terminology doesn’t usually keep me up at night. What DOES keep me up is the inability to get at a piece of data that I know should be there, but isn’t.

What I mean is that it’s fairly common that the same data point – say bandwidth – is referred to by a completely different name and location on devices from two different vendors. And maybe that doesn’t seem so weird.

But how about the same data point being different on two different types of devices from the same vendor? Still not weird?

Let’s talk about the same data point being different on the same device type from the same vendor, but two different models? Getting weird, right (not to mention annoying).

But the real kicker is when the same data point is different on two different parts of the same DEVICE.

Once you’ve run down that particular rabbit whole, you have a whole different appreciation for semantic naming. If I’m looking for CPU or bandwidth or latency or whatever, I would really REALLY like for it to be called the same thing and be found in the semantically same location.

OTel does this, and does it as a core aspect of the platform. I’m not the only one to have noticed it, either.

Several years ago, during a meeting between the maintainers of Prometheus and OpenTelemetry, an unnamed Prometheus maintainer quipped, “You know, I’m not sure about the rest of this, but these semantic conventions are the most valuable thing I’ve seen in a while.” It may sound a bit silly, but it’s also true.

Summarizing the Data

I’ll admit that OpenTelemetry is still very (VERY) shiny for me.

But I’ll also admit that the more I dig into it, the more I find to like. Hopefully this blog has given you some reasons to check out OTel, too.

* OK, I lied. 1) Perl 2) The 1967 Ford Mustang 390 GT/A and 3) deep dish from Tel Aviv Kosher Pizza in Chicago

What happened isn't the point. How you respond is.

Leon Adato — Mon, 31 Mar 2025 04:00:00 +0000

The first thing to be clear about is that “it” can happen to anybody, and it can happen at any time. The fact that it happens isn’t what you need to focus on. The details of what happened, and even how it played out, isn’t important. The truly important part is this:

How will you respond?

You may be thinking, “Why? Why would that matter? That sounds patronizing and trite.”

I don’t think so (then again, of course I don’t, I’m the one writing about it. In fact, I’ve written about it before.).

Throughout our life, things – good, bad, and somewhere in the middle – are going to happen. They’re going to happen to us, near us, and around us. There’s literally nothing anyone can do to stop things from happening. On the other hand, what is completely within our control is how we handle those things.

Aside from that, or perhaps closely related to that, is the truth that nothing about what happens reflects on who we are. But how we react and respond absolutely does.

(As a side note, remember that when you’re looking at something happening to someone else and consider making a judgement. THEY probably didn’t didn’t choose it either. But how are they responding? That’s a choice they’re making.)

A famous story present across multiple cultures tells of a queen who asks her advisors for a totem, a phrase that can give her both hope in hard times, but also keep her humble at the height of her success. It needed to be short, she added, so it could be inscribed on a ring so she could wear it and keep it in her field of vision. The advisors came back with

“This, too, shall pass.”

I think about that a lot when I’m going through low points (or high points) in my life. I’m thinking about it a lot, right now. What happened to me, today, to inspire this particular blog isn’t important. Curiously, how I responded also doesn’t matter. I’m happy to share it (and will share it, just like I did the last time but that’s not the point today.

Today, I want you to sit for a minute and imagine you get the call, and your job is over. What would you do? What would be your first, second, third step?

More interestingly, what would you do if you KNEW it was coming? Maybe you wouldn’t have done anything differently, but it’s worth considering. As I like to remind people, “There’s never a bad time to look around the airplane and make sure you know where the exits are.”

On the other hand, maybe you don’t have to imagine. Maybe that’s your lived reality right now. I’m VERY sympathetic to you, as it turns out.

In which case my advice is to skip the “what would you do” question. It’s nothing but armchair quarterbacking. The moment has passed. Sure, at some point (probably later), you can reflect on what you did, and in light of what happened, think about what you might have done differently.

But that’s for another day. Today, our job – yours and mine – is to focus on how we’re going to respond.

It Costs That Much Because it Takes Me F***ing Hours

Leon Adato — Mon, 24 Mar 2025 04:00:00 +0000

LIMITED TIME OFFER! I’m ready for my next adventure as a DevRel advocate / Technical Evangelist / IT Talespinner. If that sounds like something you need, drop me a line in email or on LinkedIn

Today’s blog title is inspired by this song. My daughter started singing it regularly after she opened her bakery. Read on for details on that story, and how it relates to observability.

I thought of it, and my daughter’s reasons for singing it, after a few responses to last week’s blog post. It touched a nerve, which was nice to see. This is an important, nuanced, and complex conversation. I believe that getting folks involved in this conversation is better for everyone.

Some of the thoughtful responses included this one:

And this one

(NOTE: click each image to read the original post)

But apparently for some folks it touched the WRONG nerve. People misunderstood the point I was trying to make. Some people (and I’m not linking to those posts) took it as a reason to bash vendors for charging anything at all. To posit that all software (and especially all monitoring software) should be free.

I wasn’t saying that monitoring tools should be free. I wasn’t even saying that vendors are charging too much. I was saying that if there’s no plan for the observability data we collect, ANY cost seems like too much. The problem is in our lack of clear goals, not the cost of services.

If there’s no plan, you can’t manage (or even estimate) the costs.

Let me explain by going back to the title of the blog, and the song that inspired it. My daughter tends to sing the catchy little ditty after one of the (unfortunately many) calls with a prospective customer, who tells her – with all the confidence of someone who’s baking expertise extends only as far as opening a box of Duncan Hines cake mix – that her cakes are way too expensive, because, and I quote: “anyone could make them.”

To her credit, my daughter (who’s run Love Plus Flour Bakery for 8 years) remains professional (and calm, which is more than I would have managed).

“Free” isn’t (always) the right answer

Monitoring and observability are ferociously difficult to get right, and the people who devote their time (and sometimes careers) to making it work – to say nothing about making it better – deserve all of our respect.

They also deserve to be paid for their efforts.

Like those folks who call my daughter’s bakery, you know who typically thinks they shouldn’t have to pay for observability? People who have only the most tenuous idea of what goes into it and how it works.

For folks at the forefront of observability, like Liz, this is nothing new.

There are those who will be quick to say, “But Leon, open source tools like Grafana are free! Why doesn’t everyone else do that?”

OK, but hear me out. While the phrase “Linux is only free if your time has no value” is far less true than when Jamie Zawinski wrote it back in 1998, the spirit of it still holds. A thing is only as free as the real total cost of it – inclusive of time, effort, and money.

Lest you think I’m working my way toward the old “Car Triangle” conundrum

…that’s NOT actually my point. It actually IS possible to have all three. My daughter can throw together a cake, or a loaf of challah, or a batch of cookies faster, better, and cheaper than she can buy them.

But of course, that’s only if you discount all the time it took her to get to this point. I won’t waste your time sharing that complete (and oft-repeated) story of the itemized $10,000 repair bill which read

Breakdown of Charges:

Turning the screw: $1

Knowing which screw to turn: $9,999

The point stands – the only cheat code for the car triangle dilemma is if you place yourself in the middle of it, bridging the gaps.

As the late 90’s commercial says, “For everything else, there’s MasterCard.”

Not All Observability Data is Equal

So back to my original point – my post last week was not trying to say observability should be free, or that vendors are charging too much. Re-iterating my main point: if there isn’t a meaningful plan for the observability data you collect, any cost is too high. The corollary is that if you DO have a meaningful plan, you will also know what it’s worth to you to collect, process, and store it.

But nothing in our lives as IT practitioners is ever as easy as coming up with a single plan for all our data and calling it a day. Different data types have different value propositions, and some of those value profiles are themselves variable based on outside factors like time of year (i.e. Black Friday).

Or what has just happened. Cribl understood this all the way back in 2022, when they framed this idea succinctly:

“As it happens, all of this high-fidelity data is completely worthless until it comes time for forensics, where it turns out to be invaluable. You have absolutely no need for such a high level of detail until the exact moment when you do need it – and you actually can’t do anything without it.”

To expand on that idea a little: I don’t think the nice folks at Cribl are saying that lo-fi data is fine for daily ops monitoring and observability, and you only need high-fi data until you’re in the middle of a retro. Instead, I think it’s akin to something my friend Alec Isaacson, Solutions Architect at Grafana, once said:

“Murphy’s law requires that you’re probably not collecting the data you need.”

Unless, of course, you are.

Cognitive (Data) Dissonance

We don’t need all the data, until we do, and in that liminal moment it goes from worthless to desperately important. We don’t want to pay anything to collect, process, or store the data. Until there’s a crisis, at which point we’re willing to spare no expense.

For the poor beleaguered monitoring engineer (and yes, I will die on the hill that this is a real job. Or should be.) this may feel like our professional Kobiyashi Maru scenario.

But it’s not. I believe there’s a way to thread this needle, and in the next blog I’m going to explore exactly that, along with how OTel fits into all of this.

Who the Hell is Going to Pay For This?

Leon Adato — Mon, 17 Mar 2025 04:00:00 +0000

LIMITED TIME OFFER! I’m ready for my next adventure as a DevRel advocate / Technical Evangelist / IT Talespinner. If that sounds like something you need, drop me a line in email or on LinkedIn

I’ve specialized in monitoring and observability for 27 years now, and I’ve seen a lot of tools and techniques come and go (RMon, anyone?); and more than a few come and stay (Rumors of the death of SNMP have been – and continue to be – greatly exaggerated.). Lately I’ve been exploring one of the more recent improvements in the space – OpenTelemetry (which I’m abbreviating to “OTel” for the remainder of this blog). I wrote about my decision to dive into OTel recently.

For the most part, I’m enjoying the journey. But there’s a problem that has existed with observability for a while now, and it’s something OTel is not helping. The title of this post hints at the issue, but I want to be more explicit. Let’s start with some comparison shopping.

Before I piss off every vendor in town, I want to be clear that these are broad, rough, high level numbers. I’ve linked to the pricing pages if you want to check the details, and I acknowledge what you see below isn’t necessarily indicative of the price you might actually pay after getting a quote on a real production environment.

New Relic charges
- 35¢ per GB for any data you send them.
- …although the pricing page doesn’t make this particularly clear
Datadog has a veritable laundry list of options, but at a high level, they charge:
- $15-$34 per host
- 60¢ – $1.22 per million netflow records
- $1.06-$3.75 per million log records
- $1.27-$3.75 per million spans
Dynatrace’s pricing page sports a list almost as long as Datadog’s but some key items:
- 15¢ per 100,00 metrics
  - plus .07¢ per Gig per day for retention
- 2¢ per gig for logs
  - plus .07¢- per gig per day to retain them
  - plus .035¢ per gig queried
- Events have the same rate as logs
- .014¢ per 1,000 spans
Grafana, which – it must be noted – is open source and effectively gives you everything for free if you’re willing to do the heavy lifting of installing and hosting. But their pricing can be summed up as:
- $8.00 for 1k metrics, (up to 1/minute)
- 50¢ per gig for logs and traces them, with 30 days retention

This list is neither exhaustive nor complete. I’ve left off a lot of vendors, not because they also don’t have consumption based pricing but because it would just be more of the same. Even with the ones above, the details here aren’t complete. Some companies not only charge for consumption (ingest), they also charge to store the data, and charge again to query the data (looking at you, New Relic). Some companies push you to pick a tier of service, and if you don’t they’ll charge you an estimated rate based on the 99th percentile of usage for the month (looking at you, Datadog).

It should surprise nobody that what appears on their pricing page isn’t even the final word. Some of these companies are, even now, looking at redefining their interpretation of the “consumption based pricing” concept that might make things even more opaque (looking at you AGAIN, New Relic).

Even with all of that said, I’m going out on a limb and stating for the record that each and every one of those price points is so low that even the word “trivial” is too big.

That is, until the production workloads meet the pricing sheet. At that point those itty bitty numbers add up to real money, and quickly.

The Plural of Anecdote

I put this question out to some friends, asking if they had real-world sticker-shock experiences. As always, my friends did not disappoint.

“I did a detailed price comparison of New Relic with Datadog a couple years ago with Fargate as the main usage. New Relic was significantly cheaper until you started shipping logs and then Datadog was suddenly 30-40% cheaper even with apm. [But] their per host cost also factors in and makes APM rather unattractive unless you’re doing something serverless. We wanted to use it on kubernetes but it was so expensive, management refused to believe the costs with services on Fargate so I was usually showing my numbers every 2-3 months.”– Evelyn Osman, Head of Platform at enmacc

“All I got is the memory of the CFOs face when he saw the bill.”– someone who prefers to remain anonymous, even though that quote is freaking epic.

And of course there’s the (now infamous, in observability circles) whodunit mystery of the $65 million Datadog bill.

The First Step is Admitting You Have a Problem

Once upon a time (by which I mean the early 2000’s), the challenge with monitoring (observability wasn’t a term we used yet) was how to identify the data we needed, and then get the systems to give up that data, and then store that data in a way that made it possible (let alone efficient) to use in queries, displays, alerts, and such.

That was where almost all the cost rested. The systems themselves were on-premises and, once the hardware was bought, effectively “free”. The result was that the accepted practice was to collect as much as possible and keep it forever. And despite the change in technology, many organizations reasoning has remained the same.

Grafana Solutions Architect Alec Isaacson points out his conversations with customers sometimes go like this:

“I collect CDM metrics from my most critical systems every 5 seconds because once, a long time ago, someone got yelled at when the system was slow and the metrics didn’t tell them why.”

Today, collecting monitoring and observability data (“telemetry”) is comparatively easy, but – both as individuals and organizations – we haven’t changed our framing of the problem. So we continue to grab every piece of data available to us. We instrument our code with every tag and span we can think of; if there’s a log message, we ship it; hardware metrics? Better grab it because it’ll provide context; If there’s network telemetry (NetFlow, VPC Flow logs, Streaming Telemetry) we suck that up too.

But we never take the time to think about what we’re going to do with it. Ms. Osman’s experience illustrates the result:

“[They] had no idea what they were doing with monitoring […] all the instrumentation and logging was enabled then there was lengthy retention “just in case”. So they were just burning ridiculous amounts of money”

To connect it to another bad behavior that we’ve (more or less) broken ourselves of: Back in the early days of “lift and shift” (often more accurately described as “lift and shit”) to the cloud, we not only moved applications wholesale; we moved it onto the biggest systems the platform offered. Why? Because in the old on-prem context you could only ask for a server once, and therefore you asked for the biggest thing you could get, in order to future-proof your investment. This decision turned out not only to be amusingly naive, it was horrifically expensive and it took everyone a few years to understand how “elastic compute” worked and to retool their applications for the new paradigm.

Likewise, it’s high time we recognize and acknowledge that we cannot afford to collect every piece of telemetry data available to us, and moreover, that we don’t have a plan for that data even if money was no object.

Admit it: Your Problem Also Has a Problem

Let me pivot to OTel for a moment. One of the key reasons – possibly THE key reason – to move to it is to remove, forever and always, the pain of vendor lock-in. This is something I explored in my last blog post and was echoed recently by a friend of mine:

OTel does solve a lot of the problems around “Oh great! now we’re trapped with vendor x and it’s going to cost us millions to refactor all this code”

as opposed to “Oh, we’re switching vendors? Cool, let me just update my endpoint…”

– Matt Macdonald-Wallace, Solutions Architect, Grafana Labs

To be very clear, OTel does an amazing job at solving this problem, which is incredible in its own right. BUT… there’s a downside to OTel that people don’t notice right away, if they notice it at all. That problem makes the previous problem even worse.

OTel takes all of your data (metrics, logs, traces, and the rest), collects it up, and sends it wherever you want it to go. But OTel doesn’t always do it EFFICIENTLY.

Example 1: log messages

Let’s take the log message below, which comes straight out of syslog. Yes, good old RFC 5424. Born in the 80’s, standardized in 2009, and the undisputed “chatty kathy” of network message protocols. I’ve seen modestly-sized networks generate upwards of 4 million syslog messages per hour. Most of it was absolutely useless drivel, mind you. But those messages had to go somewhere and be processed (or dropped) by some system along the way. It’s one of the reasons I’ve suggested a syslog and trap “filtration system” since basically forever.

Nit picking about message volume aside, there’s value in some of those messages, to some IT practitioners, some of the time. And so we have to consider (and collect) them too.

<134>1 2018-12-13T14:17:40.000Z myserver myapp 10 - [http_method="GET"; http_uri="/example"; http_version="1.1"; http_status="200"; client_addr="127.0.0.1"; http_user_agent="my.service/1.0.0"] HTTP request processed successfully

As-is that log message is 228 bytes – barely even a drop in the bucket of telemetry you collect every minute, let alone every day. But for what I’m about to do, I want a real apples-to-apples comparison, so here’s what it would look like if I JSON-ified it:

{
  "pri": 134,
  "version": 1,
  "timestamp": "2018-12-13T14:17:40.000Z",
  "hostname": "myserver",
  "appname": "myapp",
  "procid": 10,
  "msgid": "-",
  "structuredData": {
      "http_method": "GET",
      "http_uri": "/example",
      "http_version": "1.1",
      "http_status": "200",
      "client_addr": "127.0.0.1",
      "http_user_agent": "my.service/1.0.0"
        },
  "message": "HTTP request processed successfully"
}

That bumps the payload up to 336 bytes without whitespace, or 415 bytes with. Now, for comparison, here’s a sample OTLP Log message:

{
   "resource": {
      "service.name": "myapp",
      "service.instance.id": "10",
      "host.name": "myserver"
      },
   "instrumentationLibrary": {
      "name": "myapp",
      "version": "1.0.0"
      },
   "severityText": "INFO",
   "timestamp": "2018-12-13T14:17:40.000Z",
   "body": {
      "text": "HTTP request processed successfully"
       },
   "attributes": {
      "http_method": "GET",
      "http_uri": "/example",
      "http_version": "1.1",
      "http_status": "200",
      "client_addr": "127.0.0.1",
      "http_user_agent": "my.service/1.0.0"
       }
}

That (generic, minimal) message weighs in at 420 bytes (without whitespace; it’s 520 bytes all-inclusive). It’s still tiny, but even so the OTel version with whitespace is 25% bigger than the JSON-ified message (with whitespace), and more than twice as large as the original log message.

Once we start applying real-world data, things balloon even more. My point here is this: If OTel does that to every log message, these tiny costs add up quickly.

Example 2: Prometheus

It turns out that modern methods of metric management are just as susceptible to inflation.

A typical prometheus metric, formatted in JSON, is 291 bytes.
But that same metric converted to OTLP metrics format weighs in at 751 bytes.

It’s true, OTLP has a batching function that mitigates this, but that only helps with transfer over the wire. Once it arrives at the destination, many (not all, but most) vendors unbatch before storing, so it goes back to being 2.5x larger than the original message. As my buddy Josh Biggley has said,

“2.5x metrics ingest better have a fucking amazing story to tell about context to justify that cost.”

It’s Not You, OTel, It’s Us. (But It’s Also You)

If this all feels a little hyper-critical of OTel, then please give me a chance to explain. I honestly believe that OTel is an amazing advancement and anybody who’s serious about monitoring and observability needs to adopt it as a standard – that goes for users as well as vendors. The ability to emit the braid of logs, metrics, traces while maintaining its context, regardless of destination, is invaluable.

(But…) OTel was designed by (and for) software engineers. It originated in that bygone era (by which I mean “2016”) when we were still more concerned about the difficulty of getting the data than the cost of moving, processing, and storing it. OTel is, by design, biased to volume.

The joke of this section’s title notwithstanding, the problem really isn’t OTel. We really are at fault. Specifically our unhealthy relationship with telemetry. If we insist on collecting and transmitting every single data point, we have nobody to blame but ourselves for the sky-high bills we receive at the end of the month.

Does This Data Bring You Joy?

It’s easy to let your observability solution do the heavy lifting and shunt every byte of data into a unified interface. It’s easy to do if you’re a software engineer who (nominally at least) owns the monitoring and observability solutions.

It’s even easier if you’re a mere consumer of those services, an innocent bystander. Folks who fall into this category include those closely tied to a particular silo (database, storage, network, etc); or helpdesk and NOC teams who receive the tickets and provide support but aren’t involved in the instrumentation nor the tools the instrumentation is connected to; or teams with more specialized needs that nevertheless overlap with monitoring and observability, like information security.

But let’s be honest, if you’re a security engineer, how can you justify paying twice cost to ingest logs or metrics, versus the perfectly good standards that already exist and have served well for years? Does that mean you might be using more than one tool? Yes. But as I have pointed out (time and time and time and time and time and time again) there is not (and never has been, and never will be) a one-size-fits-all solution. And in most situations there’s not even a one-size-fits-MOST solution. Monitoring and observability has always been about heterogeneous implementations. The sooner you embrace that ideal, the sooner you will begin building observability ecosystems that serve the needs of you, your team, and your business.

To that end there’s a serious ROI discussion to be had before you go all in on OTel or any observability solution.

(for now)

We’ve seen the move from per seat (or interface, or chassis, or CPU) pricing to a consumption model in the marketplace in the past. And we’ve also seen technologies move back (like the way cell service moved from per-minute or per-text to unlimited data with a per-month charge). I suspect we may see a similar pendulum swing back with monitoring and observability at some time in the future. But for now, we have to contend with both the prevailing pricing system as it exists today; and with our own compulsion – born at a different point in the history of monitoring – to collect, transmit, and store every bit (and byte) of telemetry that passes beneath our nose.

Of course, cost isn’t the only factor. Performance, risk, (and more) need to be considered. But at the heart of it all is the very real need for us to start asking ourselves:

What will I do with this data?
Who will use it?
How long do I need to store it?

And of course, Who the hell is going to pay for it?