DEV Community: didiViking

One Year Inside the Engine Room of Cloud Native Days Romania

didiViking — Wed, 03 Jun 2026 12:34:11 +0000

What I learned after speaking, then stepping behind the curtains of a community conference

A year ago, I walked onto the stage at Cloud Native Days Romania as a speaker. I thought that was the main contribution: show up, deliver, leave.

It wasn’t.

The same month, I joined the organizing team of Cloud Native Days Romania, and my perspective shifted completely.

What I saw behind the scenes changed how I think about conferences, communities, and what “ecosystem growth” actually means in Eastern Europe.

From speaker to organizer: a change in mentality

Speaking at a conference is a moment. Organizing one is a system.

Once you move behind the curtain, you stop seeing talks and start seeing dependencies. The biggest shift for me was realizing this:

A conference is not an event. It’s a coordination problem across people, time zones, confidence levels, and access to opportunity.

And in Eastern Europe, that coordination carries extra weight because visibility is unevenly distributed.

CFP reviews: where signal meets bias (and why it matters)

One of the main responsibilities I took during this year was helping with the CFP reviews. On paper, it looks simple: evaluate proposals, pick the best ones. In reality, it’s much more nuanced.

You are constantly balancing: experienced speakers vs. first-time speakers, highly polished abstracts vs. high-potential ideas with weak packaging, trending topics vs. foundational knowledge the community still needs, international reach vs. local relevance.

And there’s a hidden challenge: many strong engineers underestimate themselves on paper. Some of the best talks I’ve seen would never survive a “strictly formal” CFP scoring system. That’s why review culture matters.

We didn’t just ask: Is this good?

We asked: Does this deserve a stage, and what would it take to make it ready? That second question is where communities are built.

Encouraging international speakers: visibility is not evenly distributed

Another part of my role was reaching out to international speakers and encouraging them to apply. This sounds simple, but it’s actually strategic.

Eastern European conferences often face a visibility gap: great content exists locally, but speaker pipelines are often local-heavy by default.

International speakers may not even know the event exists. So part of the job becomes outreach, storytelling, and trust-building. We need to make an effort in having more women as speakers and more professionals from underrepresented groups.

You’re not just asking someone to submit a talk — you’re saying:

“This community is worth your time.”

And that matters more than people think.

Because conferences don’t grow only through attendees. They grow through perceived relevance in the global ecosystem.

Hosting a virtual event: lowering the barrier to entry

One of the most meaningful initiatives I helped with was hosting a virtual live session aimed at:

-First-time speakers who want to apply but don’t know how

-Experienced speakers willing to share how they craft talks

-Community members curious about CFP processes

The goal was simple: reduce friction. What surprised me most was not the questions — it was the hesitation.

Many people don’t lack ideas. They lack permission structures.

They don’t know if they are “allowed” to submit.

Once you remove that invisible barrier, submissions diversify quickly. And diversity of submissions is what keeps a conference alive.

Behind the scenes: the invisible work no one sees

From the outside, a conference looks like a polished sequence of talks.

Behind the scenes, it is: multiple CFP discussions, balancing topic coverage across tracks, speaker coordination across countries and time zones, reviewing abstracts that arrive at the last minute, finding sponsors, looking the right venue, creating unique experiences for both speakers and attendees.

The rush before the event is real and this is where well organizing teams thrive. I’m happy I found the community that is both very well organized and energetic and their motivation is really inspiring for the open source and cloud native ecosystem.

The volunteers also helped me see the importance of community and understand that behind the scenes, the love for knowledge sharing is truly an valuable asset.

Merge-Forward: our community partner

Raising visibility for underrepresented groups in tech is a wonderful initiative and Merge-Forward does it brilliantly. Created last year in July and rapidly expanding, Merge-Forward hosts under its umbrella 7 working groups, which constantly thrive in community activities, mentorship and collaboration.

We hope more allies can join our mission, participate in our online monthly calls and help build a global community.

Why these events matter in Eastern Europe

Community conferences like Cloud Native Days Romania play a role that goes beyond technical talks. In Eastern Europe specifically, they function as:

Talent visibility engines: They surface engineers who may not have global platforms yet.
Knowledge distribution hubs: They bring cloud-native practices into companies that are still modernizing.
Career accelerators: Many first-time speakers later become advocates, maintainers, or engineering leaders.
Cross-border bridges: They connect engineers across countries that rarely share professional spaces otherwise. This last point is especially important.

Cloud-native systems are global, but communities are still often local. Conferences help close that gap.

What I learned after one year inside the organization

If I had to summarize the biggest lessons, they would be these:

1. Community is not passive — it is engineered.

You don’t “have” a community. You design conditions for it to form.

2. The CFP is a mentorship tool, not a filter.

The way you review submissions shapes who becomes visible.

3. First-time speakers are a high-impact investment.

They carry future ecosystems.

4. Most barriers are psychological, not technical.

People need encouragement more than instruction.

5. Conferences are infrastructure, not events.

They are part of the cloud-native stack — just social instead of technical.

How you can make a difference

I used to think speaking at conferences was the contribution.

Now I see it differently. Speaking is a contribution.

Organizing is a different one.

But enabling others to speak — especially those who never thought they could — is where the long-term impact actually sits.

And that is what makes community work in Eastern Europe feel meaningful right now: it’s not about scaling events.

It’s about scaling participation.

You can check all the Cloud Native Computing Foundation events and learn about its communities. Even if you cannot participate in person, there are online resources to help you catch up with the latest.

I’m really happy I could be part of the organizing team this year and I’m looking for your ideas on how to improve such events.

For any questions, you can find me on LinkedIn or GitHub.

Until next time, mulțumesc!

Observability Has a Data Hoarding Problem

didiViking — Mon, 11 May 2026 07:30:51 +0000

Somewhere in Greece

At the end of 2025, I gave a talk at a local meetup in Spain about something that, at the time, still felt a bit niche: Green Observability.

Most conversations around sustainability in tech tend to focus on infrastructure efficiency, cloud cost optimization, or greener hardware choices. Observability rarely enters that discussion. It is usually treated as a purely operational concern — something essential for reliability, debugging, and incident response — but not something we associate with environmental impact.

My talk explored a simple but uncomfortable idea: observability itself has a footprint.

Every metric we collect, every log we retain, every trace we store, and every dashboard query we run consumes compute, storage, and energy somewhere. At small scale, this impact feels negligible. But at the scale modern cloud-native systems operate today, observability becomes infrastructure in its own right — and infrastructure has an environmental cost.

A few months later, I expanded this topic into a talk at FOSDEM, focusing specifically on the energy and carbon footprint of modern observability systems. What stood out to me after that talk was the reaction from peers in the observability space. SREs, platform engineers, and observability practitioners came up to continue the discussion, and many strongly agreed with the core message. The feedback was consistent: telemetry growth is rarely questioned, observability systems are becoming increasingly expensive to operate, and the industry has normalized a level of data collection that often exceeds what teams actually use.

That conversation made one thing very clear to me: this is not a theoretical concern anymore. It is already happening in production systems everywhere.

The Observability Sustainability Paradox

Modern software systems are complex, distributed, and highly dynamic. Observability — collecting metrics, logs, and traces — is essential for understanding these systems. Without it, operating large-scale infrastructure would be nearly impossible.

But the same mechanisms that make observability powerful also make it expensive.

High-cardinality metrics, verbose logging, long retention periods, frequent scraping intervals, and always-on tracing pipelines significantly increase storage and compute requirements. Kubernetes environments amplify this further by generating telemetry at every layer: clusters, nodes, containers, services, and applications.

This creates what I call the observability sustainability paradox : the more data we collect to gain insight, the more energy and resources we consume to store, process, and query that data. I talked about it for the first time at Scale23x conference, in a presentation I delivered for the Cloud Native Days LA track.

At first, this trade-off feels acceptable. Visibility is critical. But over time, many systems quietly drift into excess:

metrics that are never queried,
logs retained far beyond their usefulness,
traces stored “just in case,”
dashboards nobody maintains anymore,
and pipelines that exist simply because they were never re-evaluated.

Eventually, observability stops being just a tool for understanding systems and becomes a system of its own — with its own operational overhead and environmental footprint.

Applying Green Software Principles to Observability

This is where green software thinking becomes relevant.

The goal is not to reduce observability or sacrifice reliability. The goal is to design telemetry with intention — to treat observability systems as software that should also be efficient, not just functional.

In practice, this means shifting the default mindset from “collect everything” to “collect what is useful.”

It starts with simple but important questions:

Do we actually use this metric in decision-making?
Does this dashboard influence operational behavior?
Is this alert actionable, or just informational noise?
Can we reduce retention without losing critical insight?
Are we collecting data because it is valuable, or because it is easy?

These questions are often uncomfortable because they challenge default engineering habits. But they are necessary if we want sustainable systems.

The Green Software Foundation has been driving similar thinking in broader software design: minimize waste, optimize resource usage, and consider environmental impact as a first-class constraint. Observability systems should not be excluded from that conversation.

Lessons From Practice

From a practitioner’s perspective, the most interesting insight is that reducing telemetry does not reduce observability quality — it often improves it.

In multiple systems I’ve worked with or observed, teams saw better outcomes after deliberately reducing noise in their observability stack. Not by removing critical signals, but by removing unnecessary ones.

Common improvements included:

reducing high-cardinality metrics to meaningful dimensions only,
sampling traces instead of capturing every request,
shortening retention periods for non-critical data,
simplifying dashboards that had grown organically over time,
and removing alerts that did not lead to action.

The result was consistent: less noise, faster debugging, and clearer operational signals.

One of the most counterintuitive outcomes is that observability becomes more effective when it is more constrained. When everything is monitored, nothing stands out. When telemetry is intentional, anomalies become easier to detect.

Technical Strategies for Sustainable Observability

At the technical level, sustainable observability is about rethinking the full telemetry lifecycle: generation, collection, storage, and querying.

A few practical approaches include:

Metrics should prioritize signal over dimensional explosion. High-cardinality labels should be used carefully, not by default. Logs should be structured but not overly verbose, and retention policies should reflect actual usage patterns rather than theoretical requirements.

Tracing benefits significantly from sampling strategies. Capturing every request is rarely necessary at scale, and intelligent sampling can preserve visibility into system behavior while dramatically reducing overhead.

In Kubernetes environments, there are additional optimization opportunities:

tuning scraping intervals to match actual needs,
avoiding duplicate instrumentation across layers,
aggregating metrics closer to ingestion,
reducing redundant exporters and sidecars,
and optimizing collector pipelines for efficiency.

Another emerging pattern is shifting some aggregation earlier in the pipeline, reducing the volume of data that needs to be stored or queried later. These kinds of architectural decisions often have outsized impact on both performance and cost.

Importantly, these optimizations do not just reduce resource consumption — they usually improve usability. Cleaner dashboards, faster queries, and reduced alert fatigue tend to follow naturally.

Open Source Tools Driving Change

Open source ecosystems are increasingly important in making sustainable observability practical.

Projects like OpenTelemetry provide the foundation for standardized telemetry generation and allow teams to implement sampling, filtering, and aggregation strategies consistently across systems.

At the infrastructure level, tools like Kepler are starting to make energy consumption measurable in cloud-native environments. This is a critical step, because what cannot be measured is rarely optimized.

These tools make it possible to connect observability decisions directly to resource and energy impact, rather than treating sustainability as an abstract concept.

However, most of the ecosystem is still evolving toward this awareness. Many default configurations still favor high-volume telemetry collection, and optimization is often left entirely to individual teams.

Benefits Beyond Sustainability

While environmental impact is an important motivation, sustainable observability also delivers very practical engineering benefits.

Reducing telemetry noise improves system clarity. Engineers spend less time filtering irrelevant signals and more time focusing on meaningful ones. Incident response becomes faster because dashboards are simpler and more relevant. Alert fatigue decreases because signals are more intentional.

Operational costs also drop — not just in storage, but in compute, query performance, and pipeline complexity.

In many cases, sustainability and operational excellence reinforce each other rather than conflict. Efficient observability systems tend to be easier to maintain, debug, and scale.

Culture, Measurement, and Accountability

Technology alone is not enough. Sustainable observability also requires cultural change.

Teams need to treat observability systems as first-class software systems that deserve regular review and optimization. Telemetry should not accumulate indefinitely without ownership or evaluation.

One of the most useful shifts is introducing visibility into observability itself:

How much telemetry are we generating?
What is actually being queried?
Which dashboards are still relevant?
Which metrics are never used?
What is the cost of our monitoring stack?

Even simple awareness of these questions changes behavior over time.

Sustainability also requires challenging assumptions. Many systems grow telemetry by default rather than by design. Revisiting those defaults regularly is essential to avoid long-term accumulation of waste.

The Role of the Community

The observability community has a key role to play in making this shift real.

Too often, the focus is on scale and volume rather than efficiency and signal quality. Instrumentation tends to grow faster than governance. And while tools exist to help manage telemetry, best practices around reducing waste are still not consistently applied.

The conversations I had after speaking at FOSDEM reinforced this. Many practitioners are already thinking in this direction, but there is still a gap between awareness and standard practice.

Community-driven efforts — whether through open source projects, shared patterns, or better defaults — can help close that gap. This includes encouraging smarter sampling strategies, better retention defaults, and more explicit thinking about the cost of telemetry.

If observability is becoming a core part of system architecture (which it is), then its environmental and operational footprint should be part of the design conversation from the beginning.

Rethinking What We Actually Need to Observe

I still believe observability is one of the most important disciplines in modern software engineering.

The best observability systems I’ve seen are not the ones with the most metrics or the most dashboards. They are the ones where every signal exists for a reason, where telemetry is intentional, and where engineers actively understand what they are collecting — and why.

Sustainable observability is not about collecting less for the sake of it. It is about collecting better. It is about ensuring that what we observe is meaningful enough to justify its cost.

Because in the end, the goal was never to build the biggest telemetry pipeline. The goal was to understand our systems well enough to operate them confidently and responsibly.

You can find directly my previous presentations on this topic and my entire talks’ portofolio at https://github.com/didiViking/Conferences_Talks.

Peace from Spain

Top 10 Things Not to Do at KubeCon (If You Want to Actually Enjoy It)

didiViking — Sat, 28 Mar 2026 18:21:46 +0000

Amsterdam canal photo by Diana Todea

This week, I attended my second consecutive KubeCon, KubeCon EU Amsterdam and it definitely felt like a special one.

I experienced multiple sides of the community: I attended Cloud Native Rejekts and the Maintainer Summit, spaces that are always rich in honest conversations and less crowded than the KubeCon event.

During the Maintainer Summit, I participated at the OpenTelemetry Prometheus unconference session, which I really enjoyed as it aimed to address the current pain points coming from both communities.

I spoke for the first time at both Platform Engineering Day and KubeCon itself.

At Platform Engineering Day, I co-presented “Who Designed That Platform?” with Elif Samedin, a talk that sparked great discussions around platform ownership and developer experience.

Introducing the idea of humans as a platform resonated with the audience and brought a lot of attention.

Then on KubeCon Day 1, I gave another talk on “A Simple Guide to Kubernetes Observability,” bringing this topic to a broader, more beginner friendly audience.

Day 2 was perhaps the most meaningful part of my experience: I led the first-ever CNCF Neurodiversity Community Hub session, helping amplify the mission of Merge-Forward.

The engagement, openness, and energy in that room made it clear how important these conversations are and how much space there is to grow.

One of the highlights of the week was reconnecting with the community from Cloud Native Days Romania. We ended up doing what KubeCon does best, spending time together in hallway conversations, sharing ideas, catching up, and just enjoying being part of the same ecosystem in a different city.

Moments like these remind me that the real value of these events is always the people. And of course, we’re already looking forward to welcoming everyone at the next edition of Cloud Native Days Romania, happening on May 18–19 in Bucharest.

Experiencing KubeCon from both sides — attendee and speaker — gave me a completely different perspective. And, just like in previous events, I still found myself relearning some lessons the hard way: sore feet from the wrong shoes, exhaustion from overpacked schedules.

Overall, KubeCon can be overwhelming even for experienced attendees. So instead of giving you a polished checklist of what to do, I’ll next share something far more useful: the mistakes you should avoid.

KubeCon is one of those events that feels like drinking from a firehose — thousands of attendees, hundreds of talks, endless booths, side events, and after-parties. It’s easy to fall into the trap of trying to do everything… and end up enjoying nothing.

Here are the top 10 things you should absolutely NOT do at KubeCon, if you want to survive, learn, and maybe even have a great time.

1. Don’t Wear Uncomfortable Shoes

Seriously, this is not a fashion show.

You’ll walk a lot. Between halls, booths, side events, and spontaneous hallway chats, you can easily hit 25,000–40,000 steps per day or more.

Dress for comfort, not to impress.

Nobody remembers your outfit, but your feet will remember bad decisions. It happened to me at this KubeCon so I make it the first step to take into consideration.

2. Don’t Create a Packed, Minute-by-Minute Schedule

It’s tempting to plan every talk, every meetup, every coffee.

Don’t.

KubeCon is chaotic in the best way. The most valuable moments often come from: unexpected conversations, last-minute invites, random hallway discussions.

Catching our breath with Carol Valencia, CNCF Ambassador

Leave at least 50% of your schedule open. Think of it as “strategic spontaneity.”

3. Don’t Sit in Talks All Day

Yes, the talks are great.

No, you shouldn’t attend 8 hours of them back-to-back.

Here’s the truth: most talks (if not all) are recorded, you won’t retain everything, you will miss networking opportunities.

Hand-pick a few must-see talks, then spend the rest of your time actually talking to people, visiting booths, asking those questions you’ve been waiting for this entire time.

A couple of great talks I attended were “Retroactive sampling with OpenTelemetry: cut 90% distributed tracing bandwidth usage” by Zhu Jiekun and Roman Khavronenko from VictoriaMetrics.

The second talk was “Cutting metrics traffic, cutting costs: the AZ-aware observability blueprint” by Iris Dyrmishi and Rodrigo Fior Kuntzer from Miro.

4. Don’t Ignore Networking (It’s the Real Event)

KubeCon is less about Kubernetes and more about people who build with it.

If you spend all your time consuming content, you’re missing the real value.

Don’t stick only with people you already know, avoid conversations because “it feels awkward”.

Friends from Elastic

Instead ask what people are working on, share your own challenges, follow curiosity, not status.

Reconnecting with old friends, colleagues and having honest conversations it’s worth more in reality.

5. Don’t Skip the Expo Floor

It’s easy to dismiss booths as “just vendors.” Big mistake.

The expo floor is where you can: see real-world demos, talk to engineers (not just marketers), discover tools you didn’t know you needed.

Users asking questions at the VictoriaMetrics booth

Don’t just grab swag — ask questions.

6. Don’t Forget to Explore the City

KubeCon happens in amazing cities and yet many attendees see only the conference center. Don’t be that person.

Make time to try local food, walk around neighborhoods, visit landmarks. You’ll remember that incredible local dish, that sunset walk, that random café conversation.

7. Don’t Overcommit to Evening Events

Every night, there are dozens of meetups, parties, dinners, “must-attend” gatherings.

Trying to attend all of them is a fast track to burnout.

Pick 1–2 meaningful events per night.

Then rest. Because day 2 exhaustion is very real. It happened to the best of us.

8. Don’t Be Afraid to Skip Things

You will miss things. Talks. Meetups. Conversations. That’s normal.

What’s not normal is trying to compensate by overloading yourself.

Give yourself permission to skip.

Energy is your most valuable resource at KubeCon. Saying no to after parties or dinner and just recharging your body and mind for the next day is actually a smart move.

For me, taking the Monday evening off to recharge was the best thing. I made sure to allow my body and mind to embrace the intense KubeCon that started on Tuesday.

9. Don’t Stay Only in Your Comfort Zone

It’s easy to talk only to people from your company, attend only familiar topics, stick to what you already know.

But KubeCon is the perfect place to explore adjacent technologies, hear different perspectives, challenge your assumptions.

For me, hanging out at the updates on OpenTelemetry panel was a break from the technical talks finding out what this amazing community is currently building but also new ways one can get involved.

10. Don’t Forget to Take Care of Yourself

This is the one people underestimate the most and I’ve definitely learned it the hard way.

Between long days, constant walking, irregular meals, and late nights, your body takes a hit.

Don’t ignore signs of fatigue, skip meals or hydration, assume you’ll “push through it”. And very practically: don’t forget your medicine if you need any, bring a lightweight personal kit (painkillers, band-aids, electrolytes, etc.).

It sounds simple, but it can make the difference between enjoying the event and just surviving it.

Best hang out with Adriana Villela, CNCF Ambassador and OpenTelemetry Community Manager

Final Thoughts

KubeCon is a marathon, not a sprint.

If you stay flexible, prioritize people over sessions, take care of your energy, leave space for discovery, you won’t just survive it, you’ll actually enjoy it.

And maybe, just maybe, you’ll come back with something more valuable than notes: new ideas, new connections and a fresh perspective on your work.

VictoriaMetrics playgrounds any dev should try in 2026

didiViking — Tue, 30 Dec 2025 09:39:58 +0000

VictoriaMetrics provides a rich set of public playgrounds that allow you to explore the full observability stack - metrics, logs, traces - and even query migration tools without installing or configuring anything locally. These playgrounds are backed by real VictoriaMetrics components and real data, making them ideal for:

Learning VictoriaMetrics query languages
Trying dashboards and queries interactively
Validating migration paths
Demonstrating features in talks or workshops
Experimenting with AI-driven observability via MCP

In this post, we’ll walk through each available playground, explain what it does and link to the relevant GitHub repositories behind it.

VictoriaMetrics Playground (VMUI)

🔗 https://play.victoriametrics.com/

This is the primary playground for VictoriaMetrics metrics, powered by VMUI and backed by a VictoriaMetrics cluster installation. It is available for testing the query engine, relabeling debugger, other tools and pages provided by VMUI.

This playground is the best starting point for understanding how VictoriaMetrics stores and queries metrics at scale.

GitHub repository:
https://github.com/VictoriaMetrics/VictoriaMetrics

Grafana VictoriaMetrics Playground

🔗 https://play-grafana.victoriametrics.com/

This playground provides a hosted Grafana instance preconfigured with:

VictoriaMetrics as a metrics data source
VictoriaLogs as a logs data source

Here you can:

Explore real dashboards built on top of VictoriaMetrics
See how MetricsQL and LogsQL are used in Grafana panels
Learn dashboard design and visualization best practices

It’s particularly useful if you already use Grafana and want to see how VictoriaMetrics integrates into existing workflows.

Relevant repositories:

VictoriaMetrics Grafana datasource: https://github.com/VictoriaMetrics/victoriametrics-datasource

VictoriaLogs Grafana datasource: https://github.com/VictoriaMetrics/victorialogs-datasource

VictoriaLogs Playground (LogsQL)

🔗 https://play-vmlogs.victoriametrics.com/

This playground focuses on VictoriaLogs and it is available for testing the query engine on demo logs set.
The playground demonstrates how VictoriaLogs handles high-volume log data with predictable performance and low operational overhead.

GitHub repository:
https://github.com/VictoriaMetrics/VictoriaLogs

LogQL → LogsQL Playground (Migration Assistant)

🔗 https://play-logql.victoriametrics.com/

For teams migrating from Grafana Loki, this playground provides a LogQL to LogsQL, the VictoriaLogs query language, translation tool, which automatically converts Loki queries to VictoriaLogs queries.

This significantly reduces friction when adopting VictoriaLogs in environments already using Loki.

GitHub repository:
https://github.com/VictoriaMetrics-Community/logql-to-logsql

Documentation:
https://docs.victoriametrics.com/victorialogs/logql-to-logsql/

VictoriaTraces Playground

🔗 https://play-vtraces.victoriametrics.com/

This playground showcases VictoriaTraces, the VictoriaMetrics backend for distributed tracing and it is available for testing the query engine on demo traces set.

GitHub repository:
https://github.com/VictoriaMetrics/VictoriaTraces

VMAnomaly Playground (Anomaly Detection)

🔗 https://play-vmanomaly.victoriametrics.com/metrics/vmui/

The VMAnomaly playground demonstrates automatic anomaly detection on time-series data using VictoriaMetrics. It allows you to:

Explore metrics enriched with an anomaly_score
Visualize anomalies directly in VMUI
Understand how anomaly detection integrates with MetricsQL
Learn how anomaly scores can be used for alerting (for example with vmalert)

VMAnomaly continuously analyzes metric behavior and produces anomaly scores that behave like any other time series in VictoriaMetrics.

Distribution & setup:

VMAnomaly container: https://hub.docker.com/r/victoriametrics/vmanomaly

Helm charts (including anomaly setups): https://github.com/VictoriaMetrics/helm-charts

VictoriaLogs SQL to LogsQL playground

🔗 https://play-sql.victoriametrics.com/

This playground enables you to query data from VictoriaLogs instance or just translate SQL to LogsQL without querying.

GitHub repository:
https://github.com/VictoriaMetrics/sql-to-logsql

Bonus point: VictoriaMetrics MCP server

If you are a fan of using MCP servers then you should definitely try the official VictoriaMetrics MCP Server, which enables AI-driven interaction with observability data. This MCP server allows you to use almost all read-only APIs of VictoriaMetrics, basically all the functions available in VMUI:

Querying metrics and exploring data (even drawing graphs if your client supports it)
Listing and exporting available metrics, labels, labels values and entire series
Analyzing and testing your alerting and recording rules and alerts
Showing parameters of your VictoriaMetrics instance
Exploring cardinality of your data and metrics usage statistics
Analyzing, tracing, prettifying and explaining your queries
Debugging your relabeling rules, downsampling and retention policy configurations
Integration with VictoriaMetrics Cloud

This makes it possible for MCP-compatible tools (such as Cursor, Claude Desktop, or VS Code integrations) to query and reason about observability data conversationally.

GitHub repository:
https://github.com/VictoriaMetrics-Community/mcp-victoriametrics

Video how to use VictoriaMetrics MCP server: https://www.youtube.com/watch?v=1k7xgbRi1k0

Run Your Own Local Playgrounds

In addition to these hosted playgrounds, VictoriaMetrics provides Docker and Helm-based resources you can run locally and start playing with the ecosystem:
https://hub.docker.com/u/victoriametrics
https://github.com/VictoriaMetrics/helm-charts
https://docs.victoriametrics.com/victoriametrics/quick-start/

Let me know which one was your favorite playground or if you have any feedback for improvement, ping me a comment here or you can find me on LinkedIn or GitHub.

The Unofficial Guide to Contributing to OpenTelemetry — where to look and who to talk to!

didiViking — Wed, 17 Dec 2025 09:51:50 +0000

OpenTelemetry provides the tools and standards to collect metrics, logs, and traces from applications and services. Getting started with contributions can feel overwhelming, so here is something people rarely tell you.

What People Don't Tell You

Jumping into OpenTelemetry is not just about learning the code, it is about understanding the community and the workflows behind it. Most guides focus on good first issues or setting up your local environment, but here is what you will not hear as often.
Before checking out just OpenTelemetry look at the broader ecosystem. What other tools and observability projects are out there? Are they outdated or not? Can you start contributing there as well? How do they relate to OpenTelemetry? A little bit of research and history check never hurt anyone.

Most devs don't know about this website, CLOTributor, that helps you check out good first issues from a number of Cloud Native projects. I love it!

Tips for beginners!

Start small. Contributing does not mean implementing core protocols on day one. Documentation improvements, examples, test fixes, and localization are all meaningful contributions. The codebase moves fast. APIs evolve, modules get refactored, and Collector pipelines change. Do not get discouraged if something you learned last week behaves differently today.

Community context matters. A pull request is not just code, it is discussions, design decisions and feedback from maintainers. Reading issues and pull request threads is as educational as coding itself. Your experience is your superpower. Coming from Software Engineering or developer experience gives you a fresh perspective most contributors lack. You know what patterns are practical, what documentation is confusing and which examples actually help developers adopt OpenTelemetry.

Who to Talk To

When contributing to OpenTelemetry, engage with everyone in the community, but pay particular attention to maintainers, SIG members, senior contributors and approvers. These are the people who review pull requests, set priorities and guide the direction of projects. Observing their discussions, asking thoughtful questions and seeking feedback will accelerate your learning and help your contributions align with real community needs.

Where you find them?

Useful CNCF Slack OpenTelemetry channels for new contributors:
#otel-sig-end-user, #otel-devex, #opentelemetry-new-contributors, #otel-contributor-experience, #otel-docs-localization

1. Understand the Pieces

OpenTelemetry is more than a single library, it is a set of tools and standards. SDKs exist in multiple languages including Python, Java, and Go. The Collector gathers and forwards telemetry data, and instrumentation libraries connect your frameworks and protocols. Core protocols such as OTLP, gRPC, and HTTP define how telemetry moves across systems.

If you want to go for something new and shiny in 2025, I recommend you two newer projects that are worth keeping an eye on.
OTel Injector helps automatically instrument cloud-native applications, making it easier to collect telemetry without manually modifying code.

OTel Weaver simplifies configuring and managing telemetry pipelines, reducing friction for teams setting up observability stacks.
You don't know what new and hot? Not a problem, I recommend you check out: https://opentelemetry.io/blog/2025/

Contributing to these projects allows you to influence how developers adopt OpenTelemetry in real-world environments, which is particularly valuable for SREs and platform engineers who care about scalable and maintainable monitoring. We need your developer feedback!

Direct Interviews/Feedback Sessions: The End User SIG actively seeks users for interviews to improve the project. You can reach out to them on the #otel-sig-end-user Slack channel to schedule a session.

2. Official Documentation: A Good Starting Point

The official documentation is solid but can be overwhelming if you do not know where to start. The OpenTelemetry site has quickstart guides per language, Collector setup instructions and SDK and instrumentation references. The GitHub repositories for the Collector and the Python SDK, for example, contain READMEs, contributing guides and issues labeled good first issue, which provide context and entry points for new contributors.

3. Useful Unofficial Resources: the unconventional way

Blogs, tutorials, YouTube talks, and conference sessions (e.g. KubeCon past sessions on CNCF YouTube channel) demonstrate real setups, including projects like OTel Injector. Community discussions on CNCF Slack channels often provide practical tips missing from the documentation. Contributions do not have to be code-related.
Localization of documentation, writing guides, creating sample applications, and providing feedback on instrumentation user experience are all highly valuable.

4. Setting Up a Local Sandbox

Hands-on exploration is the fastest way to learn. Start by forking and cloning the repository.

git clone https://github.com/your-username/opentelemetry-python.git cd opentelemetry-python

Set up the development environment with pip install editable for Python or go build for Go. Run tests to verify your setup.

pytest

Experiment by adding a log, tweaking instrumentation or sending data to a local Grafana instance or your preferred observability frontend platform.

5. Learn by Reading and Modifying Code

Start with small tasks such as documentation or test fixes. Study recent pull requests to learn patterns and best practices. Explore test cases as they often illustrate intended behavior clearly. Open draft pull requests for minor improvements, even for localization or example tweaks.
Be curious. Ask questions. Keep yourself motivated and challenged.

6. Expanding Your Knowledge

CNCF learning resources and courses are helpful. The Linux Foundation OpenTelemetry Certification provides a structured way to validate your understanding and gain deeper insight into the ecosystem. Hands-on projects, such as building a Collector with a sample application and Grafana pipeline, consolidate learning. Curated GitHub tutorials or Medium guides help fill gaps.

7. Making Contributions Sustainable

Sustaining contributions is about picking a focus and staying engaged. Choose a module, library, or project to focus on. Projects like OTel Injector and Weaver are particularly exciting because they address automation and pipeline management, which are pain points for many teams.

Attend SIG meetings to understand ongoing development and decisions. Share what you learn through blogs, demos, or internal talks, and review others' pull requests or mentor newcomers. Contributions that improve usability, documentation, localization, or developer experience have a long-lasting impact even if they are not code-heavy. Community is key.

Why you should listen to me?

In 2024, I was just like you. Novice, no prior Open Source or Cloud Native contribution. Then I got curious about OpenTelemetry. I joined the community and started adding my first contributions to the OpenTelemetry (OTCA) exam certification. Over the next 7–8 months, I have contributed and revised many exam questions, read official and unofficial documentation, paid attention to what my peers suggested. I showed up and I got myself motivated. With its general public release in January 2025, I became OTCA Exam developer.

Then I POC-ed OpenTelemetry at my company. It failed to reach production. But I did collaborated with developers, we instrumented code together, we made our tests, we saw the OTel potential. We provided our developer feedback to the community.

In January 2025, I contributed with my first commit and PR to the OpenTelemetry repository. Then soon followed other PRs and localization contributions. In August 2025, I created a new localization SIG for Romanian. I asked new contributors to join and mentored them.

In November 2025, at KubeCon Atlanta, the community supported us just like we supported the community all this time. Along with some of my peers, I received the OpenTelemetry 2025 Contributor Award.
In 2025, I spoke at 9 different conferences about my journey with OpenTelemetry and how important motivation is for the long run contributors in cloud native.

You can do it too. The OpenTelemetry community is extremely supportive and communicative. Be curious, ask questions, show up, diversify your contributions, stay motivated. The results won't fail to appear!

Where to find me: https://github.com/didiViking
Ask me questions: https://www.linkedin.com/in/diana-todea-b2a79968/

Article originally published on Medium, read here.

The Unofficial Guide to Contributing to OpenTelemetry — where to look and who to talk to!

didiViking — Tue, 16 Dec 2025 11:47:27 +0000

The Unofficial Guide to Contributing to OpenTelemetry — where to look and who to talk to!

Nuremberg, Germany

OpenTelemetry provides the tools and standards to collect metrics, logs, and traces from applications and services. Getting started with contributions can feel overwhelming, so here is something people rarely tell you.

What People Don’t Tell You

Before checking out just OpenTelemetry look at the broader ecosystem. What other tools and observability projects are out there? Are they outdated or not? Can you start contributing there as well? How do they relate to OpenTelemetry? A little bit of research and history check never hurt anyone.

Most devs don’t know about this website that helps you check out good first issues from a number of Cloud Native projects. I love it!

CLOTributor

Tips for beginners!

Who to Talk To

Where you find them?

Community

Useful CNCF Slack OpenTelemetry channels for new contributors:

#otel-sig-end-user, #otel-devex, #opentelemetry-new-contributors, #otel-contributor-experience, #otel-docs-localization

1. Understand the Pieces

Status

If you want to go for something new and shiny in 2025, I recommend you two newer projects that are worth keeping an eye on.

OTel Injector helps automatically instrument cloud-native applications, making it easier to collect telemetry without manually modifying code. OTel Weaver simplifies configuring and managing telemetry pipelines, reducing friction for teams setting up observability stacks.

You don’t know what new and hot? Not a problem, I recommend you check out:

2025

Direct Interviews/Feedback Sessions: The End User SIG actively seeks users for interviews to improve the project. You can reach out to them on the #otel-sig-end-user Slack channel to schedule a session.

2. Official Documentation: A Good Starting Point

The official documentation is solid but can be overwhelming if you do not know where to start.

The OpenTelemetry site has quickstart guides per language, Collector setup instructions and SDK and instrumentation references. The GitHub repositories for the Collector and the Python SDK, for example, contain READMEs, contributing guides and issues labeled good first issue, which provide context and entry points for new contributors.

3. Useful Unofficial Resources: the unconventional way

Localization of documentation, writing guides, creating sample applications, and providing feedback on instrumentation user experience are all highly valuable.

4. Setting Up a Local Sandbox

Hands-on exploration is the fastest way to learn. Start by forking and cloning the repository.

git clone https://github.com/your-username/opentelemetry-python.git
cd opentelemetry-python

Set up the development environment with pip install editable for Python or go build for Go. Run tests to verify your setup.

pytest

Experiment by adding a log, tweaking instrumentation or sending data to a local Grafana instance or your preferred observability frontend platform.

5. Learn by Reading and Modifying Code

Be curious. Ask questions. Keep yourself motivated and challenged.

6. Expanding Your Knowledge

CNCF learning resources and courses are helpful. The Linux Foundation OpenTelemetry Certification provides a structured way to validate your understanding and gain deeper insight into the ecosystem. Hands-on projects, such as building a Collector with a sample application and Grafana pipeline, consolidate learning. Curated GitHub tutorials or Medium guides help fill gaps.

7. Making Contributions Sustainable

Attend SIG meetings to understand ongoing development and decisions. Share what you learn through blogs, demos, or internal talks, and review others’ pull requests or mentor newcomers. Contributions that improve usability, documentation, localization, or developer experience have a long-lasting impact even if they are not code-heavy. Community is key.

Why you should listen to me?

In 2024, I was just like you. Novice, no prior Open Source or Cloud Native contribution. Then I got curious about OpenTelemetry. I joined the community and started adding my first contributions to the OpenTelemetry (OTCA) exam certification. Over 7–8 months, I have contributed and revised many exam questions, read official and unofficial documentation, paid attention to what my peers suggested. I showed up and I got myself motivated. With its general public release in January 2025, I became OTCA Exam developer.

In November 2025, at KubeCon Atlanta, the community supported us just like we supported the community all this time. Along with some of my peers, I received the OpenTelemetry 2025 Contributor Award.

In 2025, I spoke at 9 different conferences about my journey with OpenTelemetry and how important motivation is for the long run contributors in cloud native.

Where to find me: https://github.com/didiViking

Ask me questions: https://www.linkedin.com/in/diana-todea-b2a79968/

Observability Is a Mesh, Not a Braid

didiViking — Fri, 07 Nov 2025 05:18:30 +0000

Observability Is a Mesh, Not a Braid

Bergen, Norway. Photo by Diana Todea

We’ve often been told that observability rests on three pillars — logs, metrics, and traces — or that these strands intertwine neatly like a braid. It’s a comforting metaphor, one that suggests order and harmony among signals. But real systems aren’t orderly. They’re complex, asynchronous, and constantly shifting. Observability doesn’t behave like a braid. It behaves like a mesh.

A braid implies structure and predictability: each strand follows a path, and their intersections are fixed. But when you look closely at how telemetry behaves in distributed systems, you find anything but predictability. Data flows at different cadences, from different layers of the stack, enriched by different contexts. Metrics may emerge from log aggregation. Logs can embed trace identifiers. Traces often contain time-series attributes that behave like metrics. These aren’t distinct categories, they’re dynamic relationships.

Imagine investigating a sudden rise in latency for a payment service. The initial clue might come from a metric showing increased response times. From there, you jump into traces and find that all slow requests involve a single external API call. Tracing that further, you examine logs from the API gateway and discover retries caused by a misconfigured timeout. What connects these steps isn’t a hierarchy of signals, it’s context carried across identifiers, timestamps, and metadata. The insight doesn’t live in one signal alone but in the relationship between them.

This is why observability should be understood as a mesh of relationships, not a bundle of separate strands. Each piece of telemetry, whether it’s a counter, a span, or an event, connects through context, forming a living topology that shifts as your system evolves. The mesh isn’t made of the signals themselves but of the connections between them: shared identifiers, correlated timestamps, causal links, and metadata. When you traverse observability data, you’re not moving through pillars, you’re traversing a network of meaning.

The mesh metaphor captures how observability evolves in distributed and ephemeral environments. Consider a serverless function that spins up for milliseconds to process a queue message. It may emit a trace span without a persistent log or metric, but downstream services enrich that trace with new dimensions — latency, request size, user ID — creating new edges in the observability mesh. When a container dies or a node scales out, its telemetry doesn’t vanish into isolation; its contextual connections persist through tags, labels, and linkages.

A mesh perspective reflects how teams reason about systems in practice. When an incident unfolds, engineers don’t follow a single thread of data; they pivot. They correlate metrics with events, compare trace patterns across deployments, and examine recent configuration changes. Each hop, from metric to trace to log, is a traversal across the mesh. This movement is what makes observability powerful: it’s the ability to move seamlessly through interconnected signals to reconstruct system behavior.

Thinking of observability as a mesh shifts how we reason about its purpose. The old “pillars” describe what we collect; the mesh describes how we understand. Observability ceases to be a taxonomy of data types and becomes a topology of understanding. It’s not about piling up signals, but about weaving context between them, because insight doesn’t live inside a single metric or log line, but in the spaces between them.

A braid has order; a mesh has resilience. In an era of ephemeral infrastructure, where systems spin up and vanish in seconds, resilience is what matters. Observability as a mesh is not about permanence, it’s about adaptation. The stronger and more contextual the connections, the clearer the system’s shape becomes, even as it shifts beneath us.

From Observation to Comprehension: Living Inside the Mesh

Seeing observability as a mesh reshapes not only how we collect data, but how we think about it. It changes the developer’s relationship with telemetry, from extracting information from the system to navigating within it.

When observability is treated as a mesh, developers stop thinking in terms of silos — “check the metrics,” “look at the logs,” “open the traces” — and start thinking in paths: “Follow the context.” Instead of jumping between tools, they trace relationships: a spike in a metric leads to a trace pattern, which leads to a deployment change, which leads to a user symptom. Observability becomes a spatial experience rather than a search exercise.

In this mental model, the observability stack isn’t a dashboard full of widgets; it’s an interactive map of behavior. The developer doesn’t query isolated datasets but traverses relationships, just like moving through a graph. This shift in mindset encourages exploration over instrumentation — curiosity over configuration. Instead of asking, “Did I collect the right data?” the question becomes, “Where does this signal connect?”

For users, especially those diagnosing incidents or understanding product behavior, the mesh reframes what “visibility” means. Visibility isn’t about volume — the more logs, the better — but about connectivity: how easily can you move from one clue to another? When the mesh is strong, users don’t get lost in data. They flow through it naturally.

This way of thinking also influences design. In a mesh-oriented world, observability tools must emphasize relationships first and data types second. Interfaces should guide users through correlations: linking error rates to feature toggles, tracing slow endpoints to commits, connecting user impact to deployment history. Observability platforms become storytelling environments, narratives stitched from interconnected evidence.

The mesh democratizes observability. When context is portable, when you can follow a request’s journey from client to cache to backend without needing tribal knowledge, the barrier to understanding systems drops. Observability becomes collective memory rather than individual expertise.

Ultimately, the mesh mindset moves observability beyond diagnostics into dialogue. Developers no longer interrogate their systems; they converse with them. Each relationship — between signals, between components, between intent and outcome — becomes part of an evolving conversation about how the system behaves. The observability mesh is the language of that conversation.

Conclusion

Observability is not static infrastructure, it’s a living, breathing topology that tells the story of a system in motion. Seeing it as a mesh allows us to move beyond classification toward comprehension — to stop counting pillars and start tracing connections.

Note: The term “observability as a mesh” has been mentioned before, such as in a 2024 article by SigNoz. This piece builds on that phrasing to develop a new conceptual model—treating the mesh not just as interconnected data, but as an adaptive, cognitive topology that reflects how engineers think, reason, and interact with systems.

OTel Me More on Traces: introducing VictoriaMetrics’ Trace Analyzer

didiViking — Thu, 21 Aug 2025 09:47:17 +0000

Tracing is an essential part of modern observability, helping developers understand how requests flow through distributed systems. OpenTelemetry (OTel) has become the de facto standard for collecting traces across services, and VictoriaMetrics’ UI now includes a powerful Trace Analyzer that provides detailed execution traces of queries. These traces show how VictoriaMetrics queries are processed internally, highlighting stages, timings, and resource usage so you can turn raw query execution data into actionable performance insights. In this article, we’ll explore how the Trace Analyzer works, how to use it, why it matters for your monitoring stack and how you can combine it with OTel to get both system-wide spans and deep query-level traces.

Why Tracing Matters

In complex architectures, latency issues and failures rarely occur in isolation. Distributed tracing helps answer critical questions:

Which service is slowing down my request?
Where are errors originating in a multi-service call chain?
How does traffic affect the performance of each component?

OTel provides a standardized way to instrument your applications and collect trace data. However, collecting traces is just the first step, analyzing them efficiently is where the real value comes in.

Introducing VictoriaMetrics Trace Analyzer

VictoriaMetrics’ Trace Analyzer is designed to make query trace exploration intuitive and fast. With it, you can:

Visualize query execution as an interactive tree, showing each evaluation step and how long it took.
Inspect details such as time ranges, step size, number of matched series, and points processed.
Identify bottlenecks in queries (e.g. heavy rollups, large series scans, or cache misses).
Combine Trace Analyzer with OTel-collected spans in your stack: use OTel for distributed request flows and Trace Analyzer for deep insights into how VictoriaMetrics executes your queries.

Before diving into setup, let’s get a better understanding of VictoriaMetrics’ architecture and where Trace Analyzer fits.

Components of a VictoriaMetrics cluster explained

In VMUI (VictoriaMetrics Native UI), you’ll discover powerful, unique tools such as Trace Analyzer, Trace Query, and Query Analyzer, along with features that make exploring and debugging metrics a breeze. These include raw metrics exploration, cardinality analysis, top and active queries inspection, a WITH expressions playground, metric relabeling, downsampling, and retention filter debuggers.

Getting Started

Install either VictoriaMetrics single-node or cluster version. For this tutorial, I’m using VictoriaMetrics single-node version and the setup will clearly be a lot easier. In a VictoriaMetrics single-node setup, we do not need a separate vmagent container because the single-node binary already includes a built-in Prometheus-compatible scraping engine. All Prometheus-compatible targets (like node_exporter or application endpoints) can be discovered and scraped directly using a scrape.yaml configuration file, which is mounted into the VictoriaMetrics container. This file defines all the jobs, endpoints, and labels needed for metrics collection, so adding vmagent would be redundant and could cause duplicate scrapes.
I installed VictoriaMetrics single-node and defined all static scrape jobs in a scrape.yaml file.

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['node-exporter:9100']

  - job_name: 'standard-app'
    static_configs:
      - targets: ['demo-app:8083']

  - job_name: 'highcard'
    static_configs:
      - targets: ['demo-highcard-app:8084']

  - job_name: 'traffic-skew'
    static_configs:
      - targets: ['demo-traffic-skew-app:8085']

Pull and run the OTel Collector, I used the docker image setup for otel/opentelemetry-collector-contrib, and I defined a docker-compose.yaml file where I added my entire setup. Below my docker-compose.yaml file which has a Grafana OSS local setup, and some basic apps to generate traffic and metrics.

services:
  victoria-metrics:
    image: victoriametrics/victoria-metrics:v1.122.0
    ports:
      - "18428:8428"

  grafana:
    image: grafana/grafana:11.1.0
    ports:
      - "13000:3000"
    environment:
      - GF_SECURITY_ADMIN_USER=[redacted]
      - GF_SECURITY_ADMIN_PASSWORD=[redacted]
    volumes:
      - ./grafana-provisioning:/etc/grafana/provisioning
    depends_on:
      - victoria-metrics

  node-exporter:
    image: prom/node-exporter:v1.7.0
    ports:
      - "9101:9100"

  demo-app:
    build:
      context: ./demo-app
    environment:
      - PORT=8083
      - LABEL_COUNT=5
    ports:
      - "8083:8083"

  loadgen-demo-app:
    image: python:3.12-alpine
    depends_on:
      - demo-app
    command: >
      sh -c "pip install --no-cache-dir requests &&
             python -u -c 'import requests, time; 
             while True: requests.get(\"http://demo-app:8083\"); time.sleep(0.1)'"

  demo-highcard-app:
    build:
      context: ./demo-highcard-app
    environment:
      - PORT=8084
      - LABEL_COUNT=50
    ports:
      - "8084:8084"

  loadgen-demo-highcard-app:
    image: python:3.12-alpine
    depends_on:
      - demo-highcard-app
    command: >
      sh -c "pip install --no-cache-dir requests &&
             python -u -c 'import requests, time; 
             while True: requests.get(\"http://demo-highcard-app:8084\"); time.sleep(0.1)'"

  demo-traffic-skew-app:
    build:
      context: ./demo-traffic-skew-app
    environment:
      - PORT=8085
      - LABEL_COUNT=5
    ports:
      - "8085:8085"

  loadgen-demo-traffic-skew-app:
    image: python:3.12-alpine
    depends_on:
      - demo-traffic-skew-app
    command: >
      sh -c "pip install --no-cache-dir requests &&
             python -u -c 'import requests, time; 
             while True: requests.get(\"http://demo-traffic-skew-app:8085\"); time.sleep(0.1)'

  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.132.0
    command: ["--config=/etc/otel-collector/config.yaml"]
    volumes:
      - ./otel-collector/config.yaml:/etc/otel-collector/config.yaml:ro

    ports:
      - "8889:8889"  
      - "4317:4317"  
      - "4318:4318"
    depends_on:
      - victoria-metrics
      - demo-app
      - demo-highcard-app
      - demo-traffic-skew-app
      - node-exporter

However, we still use an OTel scrape job in the collector’s config because it allows the OTel Collector to expose its own internal metrics (like otelcol_exporter_send_failed_metric_points_total) to VictoriaMetrics. Without this job, these important telemetry metrics from the collector itself would not be visible. Essentially: scrape.yaml handles all external targets, while the OTel scrape job handles self-monitoring of the collector. Below you can view my otel-collector config.yaml file, where metricsql-lab-victoria-metrics-1 is the hostname used in this demo setup. In this case, the otel-collector config.yaml file will be stored in a separate folder from where my victoriametrics and the metrics apps are hosted.

receivers:
  otlp:
    protocols:
      grpc:
        endpoint: "0.0.0.0:4317"
      http:
        endpoint: "0.0.0.0:4318"

  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          static_configs:
            - targets: ['0.0.0.0:8888']

  filelog:
    include: ["/logs/*.log"]
    start_at: beginning

processors:
  resourcedetection:
    detectors: [system]
  batch:
    timeout: 5s

exporters:
  prometheusremotewrite:
    endpoint: "http://metricsql-lab-victoria-metrics-1:8428/api/v1/write"
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 0

  prometheus:
    endpoint: "0.0.0.0:8889"

service:
  telemetry:
    metrics:
      level: basic

  pipelines:
    metrics:
      receivers: [otlp, prometheus]
      processors: [resourcedetection, batch]
      exporters: [prometheus, prometheusremotewrite]

Ensure the OTel Collector is on the same Docker network as VictoriaMetrics and the demo apps. You might run into different errors, depending on how complex your setup is, and if one of those errors is network related, you might want to troubleshoot it with:

docker network ls
docker inspect network <your-network>

If the otel collector doesn’t show in the same network with the rest of your services, simply add it to the same network, reboot your otel collector service and you’re good to test.

docker network connect <your-network> otelcol
docker restart <otelcol>
docker ps -a
docker logs -f <otelcol>

Switch to VMUI: you can go directly into the http://localhost:8428 or whatever port you have defined for it.

What I like to do immediately is go to Explore>Explore Cardinality. This is where the juicy part starts happening and this view actually helps me troubleshoot cardinality.

Looking at the metric names with highest number of series I can drill down into the metric name. In this case, I’m searching for otelcol-contrib, so I’ll check the labels section, job and drill down there.

Done. Now I’m going even further to look at the metrics for this job.

This panel helps me understand which metrics have the highest cardinality and which have the lowest. To learn how to use the Cardinality Explorer properly and be more efficient in your troubleshooting, then please read this article.

For example this panel shows us OTel Collector’s own internal metrics, which are very useful in providing us the necessary info to monitor the OTel collector’s health. We can check if:

the collector is receiving data from all configured sources (otelcol_receiver_accepted_spans_total, otelcol_receiver_received_metrics_total)
there are any exporters failing to send metrics or traces to the backend (otelcol_exporter_send_failed_metric_points_total)
how fast is the collector processing incoming telemetry (otelcol_processor_accepted_spans_total, otelcol_processor_dropped_spans_total)
there are any backlogs or bottlenecks forming in any pipeline stages

A concrete example of this metric otelcol_exporter_prometheusremotewrite_consumers will let me know how many “consumer pipelines” are connected to that exporter at a given moment and verify that the metrics pipelines are actually connected to the remote write backend and not idle.

We can drill down into each query by exploring the Query and its capabilities. Using the query {job="otelcol-contrib} I can activate Trace query by enabling the button below it and executing the query again to get the results.

When you enable query tracing in VictoriaMetrics, each line in the output represents a step in the query’s execution. The trace shows how long the step took, which API endpoint was called and what kind of operation was performed. You’ll also see the time range that was evaluated, the resolution of the query, and the actual expression being run. At the end of the line, VictoriaMetrics reports how many time series matched the query and how many data points were processed. Taken together, these lines form a detailed breakdown of the query lifecycle, making it easier to understand how the system executed your request and where most of the work happened.

Let’s save one of these traces to JSON and explore the Trace Analyzer.

Upload the trace JSON file into Trace Analyzer, then expand all nodes to explore details.

Once you upload a trace JSON into VictoriaMetrics’ Trace Analyzer , you get more than just raw query execution lines, it offers interactive exploration and analysis : you can see the query broken down into a tree of operations , where each node shows the type of step (e.g., evaluation, rollup, merge), how long it took, and how many series and points it processed. You can expand or collapse nodes to focus on the steps that matter most and quickly identify bottlenecks or expensive parts of your query. It also highlights cache hits vs full computation , so you can see whether repeated queries benefit from caching.

Additionally, the interface lets you filter, sort, and navigate through large traces, making it easier to understand complex queries at a glance. Essentially, the Trace Analyzer transforms a static JSON trace into a visual, interactive breakdown that makes query performance and optimization much more intuitive.

Beyond visualizing queries, Trace Analyzer helps you optimize performance by revealing which steps take the most time or process the most series. It lets you monitor cache effectiveness , compare runs to spot regressions, and provides actionable insights for query tuning, infrastructure adjustments, and team education.

What’s Ahead: VictoriaTraces and the Bigger Observability Picture

While Trace Analyzer gives deep insights into how VictoriaMetrics executes individual queries, VictoriaTraces takes observability to the next level. VictoriaTraces is designed to handle distributed tracing at scale , collecting and storing OpenTelemetry spans across services and applications. It complements VictoriaMetrics by focusing on end-to-end request flows , while Trace Analyzer focuses on query-level execution details.

Together, they provide a full-spectrum view of your observability stack : VictoriaTraces shows how requests move through your distributed system, helping you detect latency or bottlenecks across services, while Trace Analyzer lets you dig into the performance of your queries themselves , identifying expensive steps, cache behavior, and series volume. By combining the two, developers and SREs can correlate application-level traces with database/query-level insights , making performance optimization more precise and actionable.

This integration strengthens the overall ecosystem, enabling teams to move from detection to diagnosis to optimization faster, all within a unified, efficient monitoring environment.

Ready to explore more and engage with the community?

Have questions for me? Give me a shout on:

Github

Bluesky: @didiviking.bsky.social

X: @dianavtodea

Mastodon: @dianatodea