Last Thursday in Denver, US, OTel Community Day event took place as part of the Open Observability Summit and it was a blast. I had the chance to participate with a lightning talk and learn about all the latest releases within the Open Observability realm.
Below, you can find the summary based on the sessions I participated at, the official videos will be released on the CNCF Youtube channel in a couple of weeks. The OTel Community Day had two parallel tracks and some of the presentations are already uploaded on the schedule page.
Keynotes summary:
Margaret Dawson from Chronosphere’s opening speech resonated with me as we “should take observability seriously right from the beginning when we start defining our infrastructure not after we already have built platforms or tools”. Moreover, observability should be considered as an abstraction layer and we should definitely focus on “standardizing the open” source.
Austin Parker from Honeycomb’s opening notes paralleled Margaret’s message, as OpenTelemetry is becoming an observability standard, and we should all start contributing to this open source project.
Dotan Horovits shared useful information on OpenSearch new releases, marking an important mark for version 3.1 which is already out, query insights for observability use cases, cross cluster search for traces, PPL for nested JSON, NLP querying and many more. OpenSearch Observability SIG is looking for contributors, so no need to think twice, check out their Slack channel and start contributing.
Gordon Radlein from Datadog pointed out how LLMs are doing analysis faster, but we still should focus on data, semantic conventions and open source standards such as the one OpenTelemetry is proposing.
Let’s get right into the talks:
Yuri Oliveira Sa from Olly Garden and Jamie Danielson from Honeycomb took us out for a nice comparison between traces and spans and their important moments in life. I really liked how each significant moment of a life of span or trace was linked to processing, exporting, having attributes and making usage of Otel collector. It was very creative and I’m sure the video of this talk will get you more excited about traces and spans.
Henrik Rexed from Dynatrace had two brilliant talks. One was about performance benchmark comparing Fluentbit with OpenTelemetry, sharing how he benchmarked logging, traces and metrics, performance for each of these two projects. I’m not going to spoil the content of his talks, but overall Fluentbit should take care of adding a LoadBalancer exporter and a metrics converter while OpenTelemetry’s myriad of pipelines is not necessarily doing them any favors. The other talk was all about tail and probabilistic sampling, you should definitely watch both recordings for an in-depth walkthrough into this content.
Endre Sara from Causely had a very interesting talk, which invited us to focus on semantic conventions as ontology. It resonated with a lot of discussions going on lately around OpenTelemetry instrumentation: how we should not log all telemetry data and should focus more on the relationships between data. Very nice touch on LD_PREload for OTel agent and epbf instrumentation shout out. You can check more about #otel-injector and #otel-epbf-instrumentation SIGs and start contributing.
Hanson Ho, from Embrace, approver for OTel android SIG, gave a great talk pointing out that there is a lack of KMP support at the moment, and there is currently work undergoing for an Kotlin SDK. Hopefully in the future this will be shared with the open source community and more progress will get done. Be sure to check out the #otel-android SIG and start contributing.
Braydon Kains from Google shared with us his journey on building Distrogen, a tool for generating OTel Collector distributions built by OCB. Braydon is a member of the #otel-system-metrics SIG, be sure to give him a ping on the CNCF Slack channel if you are interested in finding more about Distrogen or if you want to contribute to this SIG.
Budhaditya Bhattacharya from Tyk gave an exciting talk about the 5 major Don’ts in OpenTelemetry, based on his experience so far in building an API gateway level within his organization. To summarize, we should NOT:
- start without a plan,
- collect everything, this will lead to a risk in high spending (always have a sampling strategy),
- disregard security and privacy,
- use custom attributes off the date (we should prioritize OTel semantic conventions),
- forget context propagations.
Jason Plumb from Splunk gave an excellent talk about how to manage the instrumentation overhead. I really wish I have knew all about this when I POC-ed OpenTelemetry and share this information with my team as lots of pain points we had resonated deeply with this talk. Take-aways from this talk?
- Instrumentation is not awful and is not free.
- Latency percentages are lies.
- You should carefully consider other factors that cause instrumentation overhead: instrumentation itself, instrumentation configuration, service usage, data volumes, environment.
- Always test and measure in a controlled environment, which is as close as possible to your production, and with similar workloads.
I’m not going to spoil ALL of this talk’s content, you should definitely look out for this recording, it’s a must see.
Lauren Roshore from Spotify walked us through their journey with open source observability and how legacy systems bled their performance and reliability dry. Luckily they focused on a mix of open source tools and time-series database, which came to their help in the migration phase. Choosing VictoriaMetrics as time-serie database and adopting VMAlert with in-house tooling was a great move, leading to great cost savings, 10x faster query speeds, reliable alerting and improved dashboards. Their long term plan is to focus on OpenTelemetry for logging and tracing and aligning this strategy with their existing OSS stack. What I particularly liked about this talk was the creative “Alert Plumber System”, be sure to check it out in the recording.
Yang Song from Datadog told us all about OTel collector, deep dived into its architecture and shared tips and tricks that are super useful: e.g. look out for OTel Registry for easier discoverability. Yang is part of the #otel-collector SIG, be sure to reach out to him via Slack if you have more questions or if you are interested in contributing.
There were obviously a lot more interesting and excellent talks I missed as the conference had two tracks, but what is certain all the new and good was mentioned.
Just by talking with Bindplane, of the sponsors of this event, I learnt more about their contributions to OTel Supervisor, with #otel-opamp as SIG. Open Agent Management Protocol (OpAMP) is a network protocol for remote management of large fleets of data collection Agents and you can learn more about the OTel Supervisor on this github page.
Many other topics were discussed such as LLMObservability and how GenerativeAI is hepling with traces visibility and log analysis. Be on the watch out for SQL commenter (an observability technique) from Google, Lin Lin and Tammy Baylis from SolarWinds gave us a great intro and demo, Chris Gray and Cijo Thomas from Microsoft gave a great talk about OTLP and be sure to watch #otel-dynamic-telemetry SIG for more, GrafanaLabs gave excellent intro talks on OpenTelemetry and JavaScript demos, and many other talks which I’m not really doing any justice, so you have to watch the recordings yourself.
My lightning talk was about the first contributions to OpenTelemetry and how you should aim to diversify them, moving from code to no code contributions (and vice-versa), staying in touch with local CNCF meetups and #otel-docs-locatization SIG. I’m particularly proud about the progress on the OTel Spanish docs translation and we really need more Spanish speakers to join our efforts. Please feel free to join this SIG (#otel-localization-es) and make OTel docs available into Spanish and other languages.
My talks are available at https://github.com/didiViking/Conferences_Talks, be sure to read them and follow up with me if you have any questions or improvement suggestions.
Happy OTel-ing everyone and until next time, stay curious!
Quick update: all the videos from OTel Day are already released, happy viewing 😊
Top comments (0)