DEV Community: didiViking

Observability Has a Data Hoarding Problem

didiViking — Mon, 11 May 2026 07:30:51 +0000

Somewhere in Greece

At the end of 2025, I gave a talk at a local meetup in Spain about something that, at the time, still felt a bit niche: Green Observability.

Most conversations around sustainability in tech tend to focus on infrastructure efficiency, cloud cost optimization, or greener hardware choices. Observability rarely enters that discussion. It is usually treated as a purely operational concern — something essential for reliability, debugging, and incident response — but not something we associate with environmental impact.

My talk explored a simple but uncomfortable idea: observability itself has a footprint.

Every metric we collect, every log we retain, every trace we store, and every dashboard query we run consumes compute, storage, and energy somewhere. At small scale, this impact feels negligible. But at the scale modern cloud-native systems operate today, observability becomes infrastructure in its own right — and infrastructure has an environmental cost.

A few months later, I expanded this topic into a talk at FOSDEM, focusing specifically on the energy and carbon footprint of modern observability systems. What stood out to me after that talk was the reaction from peers in the observability space. SREs, platform engineers, and observability practitioners came up to continue the discussion, and many strongly agreed with the core message. The feedback was consistent: telemetry growth is rarely questioned, observability systems are becoming increasingly expensive to operate, and the industry has normalized a level of data collection that often exceeds what teams actually use.

That conversation made one thing very clear to me: this is not a theoretical concern anymore. It is already happening in production systems everywhere.

The Observability Sustainability Paradox

Modern software systems are complex, distributed, and highly dynamic. Observability — collecting metrics, logs, and traces — is essential for understanding these systems. Without it, operating large-scale infrastructure would be nearly impossible.

But the same mechanisms that make observability powerful also make it expensive.

High-cardinality metrics, verbose logging, long retention periods, frequent scraping intervals, and always-on tracing pipelines significantly increase storage and compute requirements. Kubernetes environments amplify this further by generating telemetry at every layer: clusters, nodes, containers, services, and applications.

This creates what I call the observability sustainability paradox : the more data we collect to gain insight, the more energy and resources we consume to store, process, and query that data. I talked about it for the first time at Scale23x conference, in a presentation I delivered for the Cloud Native Days LA track.

At first, this trade-off feels acceptable. Visibility is critical. But over time, many systems quietly drift into excess:

metrics that are never queried,
logs retained far beyond their usefulness,
traces stored “just in case,”
dashboards nobody maintains anymore,
and pipelines that exist simply because they were never re-evaluated.

Eventually, observability stops being just a tool for understanding systems and becomes a system of its own — with its own operational overhead and environmental footprint.

Applying Green Software Principles to Observability

This is where green software thinking becomes relevant.

The goal is not to reduce observability or sacrifice reliability. The goal is to design telemetry with intention — to treat observability systems as software that should also be efficient, not just functional.

In practice, this means shifting the default mindset from “collect everything” to “collect what is useful.”

It starts with simple but important questions:

Do we actually use this metric in decision-making?
Does this dashboard influence operational behavior?
Is this alert actionable, or just informational noise?
Can we reduce retention without losing critical insight?
Are we collecting data because it is valuable, or because it is easy?

These questions are often uncomfortable because they challenge default engineering habits. But they are necessary if we want sustainable systems.

The Green Software Foundation has been driving similar thinking in broader software design: minimize waste, optimize resource usage, and consider environmental impact as a first-class constraint. Observability systems should not be excluded from that conversation.

Lessons From Practice

From a practitioner’s perspective, the most interesting insight is that reducing telemetry does not reduce observability quality — it often improves it.

In multiple systems I’ve worked with or observed, teams saw better outcomes after deliberately reducing noise in their observability stack. Not by removing critical signals, but by removing unnecessary ones.

Common improvements included:

reducing high-cardinality metrics to meaningful dimensions only,
sampling traces instead of capturing every request,
shortening retention periods for non-critical data,
simplifying dashboards that had grown organically over time,
and removing alerts that did not lead to action.

The result was consistent: less noise, faster debugging, and clearer operational signals.

One of the most counterintuitive outcomes is that observability becomes more effective when it is more constrained. When everything is monitored, nothing stands out. When telemetry is intentional, anomalies become easier to detect.

Technical Strategies for Sustainable Observability

At the technical level, sustainable observability is about rethinking the full telemetry lifecycle: generation, collection, storage, and querying.

A few practical approaches include:

Metrics should prioritize signal over dimensional explosion. High-cardinality labels should be used carefully, not by default. Logs should be structured but not overly verbose, and retention policies should reflect actual usage patterns rather than theoretical requirements.

Tracing benefits significantly from sampling strategies. Capturing every request is rarely necessary at scale, and intelligent sampling can preserve visibility into system behavior while dramatically reducing overhead.

In Kubernetes environments, there are additional optimization opportunities:

tuning scraping intervals to match actual needs,
avoiding duplicate instrumentation across layers,
aggregating metrics closer to ingestion,
reducing redundant exporters and sidecars,
and optimizing collector pipelines for efficiency.

Another emerging pattern is shifting some aggregation earlier in the pipeline, reducing the volume of data that needs to be stored or queried later. These kinds of architectural decisions often have outsized impact on both performance and cost.

Importantly, these optimizations do not just reduce resource consumption — they usually improve usability. Cleaner dashboards, faster queries, and reduced alert fatigue tend to follow naturally.

Open Source Tools Driving Change

Open source ecosystems are increasingly important in making sustainable observability practical.

Projects like OpenTelemetry provide the foundation for standardized telemetry generation and allow teams to implement sampling, filtering, and aggregation strategies consistently across systems.

At the infrastructure level, tools like Kepler are starting to make energy consumption measurable in cloud-native environments. This is a critical step, because what cannot be measured is rarely optimized.

These tools make it possible to connect observability decisions directly to resource and energy impact, rather than treating sustainability as an abstract concept.

However, most of the ecosystem is still evolving toward this awareness. Many default configurations still favor high-volume telemetry collection, and optimization is often left entirely to individual teams.

Benefits Beyond Sustainability

While environmental impact is an important motivation, sustainable observability also delivers very practical engineering benefits.

Reducing telemetry noise improves system clarity. Engineers spend less time filtering irrelevant signals and more time focusing on meaningful ones. Incident response becomes faster because dashboards are simpler and more relevant. Alert fatigue decreases because signals are more intentional.

Operational costs also drop — not just in storage, but in compute, query performance, and pipeline complexity.

In many cases, sustainability and operational excellence reinforce each other rather than conflict. Efficient observability systems tend to be easier to maintain, debug, and scale.

Culture, Measurement, and Accountability

Technology alone is not enough. Sustainable observability also requires cultural change.

Teams need to treat observability systems as first-class software systems that deserve regular review and optimization. Telemetry should not accumulate indefinitely without ownership or evaluation.

One of the most useful shifts is introducing visibility into observability itself:

How much telemetry are we generating?
What is actually being queried?
Which dashboards are still relevant?
Which metrics are never used?
What is the cost of our monitoring stack?

Even simple awareness of these questions changes behavior over time.

Sustainability also requires challenging assumptions. Many systems grow telemetry by default rather than by design. Revisiting those defaults regularly is essential to avoid long-term accumulation of waste.

The Role of the Community

The observability community has a key role to play in making this shift real.

Too often, the focus is on scale and volume rather than efficiency and signal quality. Instrumentation tends to grow faster than governance. And while tools exist to help manage telemetry, best practices around reducing waste are still not consistently applied.

The conversations I had after speaking at FOSDEM reinforced this. Many practitioners are already thinking in this direction, but there is still a gap between awareness and standard practice.

Community-driven efforts — whether through open source projects, shared patterns, or better defaults — can help close that gap. This includes encouraging smarter sampling strategies, better retention defaults, and more explicit thinking about the cost of telemetry.

If observability is becoming a core part of system architecture (which it is), then its environmental and operational footprint should be part of the design conversation from the beginning.

Rethinking What We Actually Need to Observe

I still believe observability is one of the most important disciplines in modern software engineering.

The best observability systems I’ve seen are not the ones with the most metrics or the most dashboards. They are the ones where every signal exists for a reason, where telemetry is intentional, and where engineers actively understand what they are collecting — and why.

Sustainable observability is not about collecting less for the sake of it. It is about collecting better. It is about ensuring that what we observe is meaningful enough to justify its cost.

Because in the end, the goal was never to build the biggest telemetry pipeline. The goal was to understand our systems well enough to operate them confidently and responsibly.

You can find directly my previous presentations on this topic and my entire talks’ portofolio at https://github.com/didiViking/Conferences_Talks.

Peace from Spain

VictoriaMetrics playgrounds any dev should try in 2026

didiViking — Tue, 30 Dec 2025 09:39:58 +0000

VictoriaMetrics provides a rich set of public playgrounds that allow you to explore the full observability stack - metrics, logs, traces - and even query migration tools without installing or configuring anything locally. These playgrounds are backed by real VictoriaMetrics components and real data, making them ideal for:

Learning VictoriaMetrics query languages
Trying dashboards and queries interactively
Validating migration paths
Demonstrating features in talks or workshops
Experimenting with AI-driven observability via MCP

In this post, we’ll walk through each available playground, explain what it does and link to the relevant GitHub repositories behind it.

VictoriaMetrics Playground (VMUI)

🔗 https://play.victoriametrics.com/

This is the primary playground for VictoriaMetrics metrics, powered by VMUI and backed by a VictoriaMetrics cluster installation. It is available for testing the query engine, relabeling debugger, other tools and pages provided by VMUI.

This playground is the best starting point for understanding how VictoriaMetrics stores and queries metrics at scale.

GitHub repository:
https://github.com/VictoriaMetrics/VictoriaMetrics

Grafana VictoriaMetrics Playground

🔗 https://play-grafana.victoriametrics.com/

This playground provides a hosted Grafana instance preconfigured with:

VictoriaMetrics as a metrics data source
VictoriaLogs as a logs data source

Here you can:

Explore real dashboards built on top of VictoriaMetrics
See how MetricsQL and LogsQL are used in Grafana panels
Learn dashboard design and visualization best practices

It’s particularly useful if you already use Grafana and want to see how VictoriaMetrics integrates into existing workflows.

Relevant repositories:

VictoriaMetrics Grafana datasource: https://github.com/VictoriaMetrics/victoriametrics-datasource

VictoriaLogs Grafana datasource: https://github.com/VictoriaMetrics/victorialogs-datasource

VictoriaLogs Playground (LogsQL)

🔗 https://play-vmlogs.victoriametrics.com/

This playground focuses on VictoriaLogs and it is available for testing the query engine on demo logs set.
The playground demonstrates how VictoriaLogs handles high-volume log data with predictable performance and low operational overhead.

GitHub repository:
https://github.com/VictoriaMetrics/VictoriaLogs

LogQL → LogsQL Playground (Migration Assistant)

🔗 https://play-logql.victoriametrics.com/

For teams migrating from Grafana Loki, this playground provides a LogQL to LogsQL, the VictoriaLogs query language, translation tool, which automatically converts Loki queries to VictoriaLogs queries.

This significantly reduces friction when adopting VictoriaLogs in environments already using Loki.

GitHub repository:
https://github.com/VictoriaMetrics-Community/logql-to-logsql

Documentation:
https://docs.victoriametrics.com/victorialogs/logql-to-logsql/

VictoriaTraces Playground

🔗 https://play-vtraces.victoriametrics.com/

This playground showcases VictoriaTraces, the VictoriaMetrics backend for distributed tracing and it is available for testing the query engine on demo traces set.

GitHub repository:
https://github.com/VictoriaMetrics/VictoriaTraces

VMAnomaly Playground (Anomaly Detection)

🔗 https://play-vmanomaly.victoriametrics.com/metrics/vmui/

The VMAnomaly playground demonstrates automatic anomaly detection on time-series data using VictoriaMetrics. It allows you to:

Explore metrics enriched with an anomaly_score
Visualize anomalies directly in VMUI
Understand how anomaly detection integrates with MetricsQL
Learn how anomaly scores can be used for alerting (for example with vmalert)

VMAnomaly continuously analyzes metric behavior and produces anomaly scores that behave like any other time series in VictoriaMetrics.

Distribution & setup:

VMAnomaly container: https://hub.docker.com/r/victoriametrics/vmanomaly

Helm charts (including anomaly setups): https://github.com/VictoriaMetrics/helm-charts

VictoriaLogs SQL to LogsQL playground

🔗 https://play-sql.victoriametrics.com/

This playground enables you to query data from VictoriaLogs instance or just translate SQL to LogsQL without querying.

GitHub repository:
https://github.com/VictoriaMetrics/sql-to-logsql

Bonus point: VictoriaMetrics MCP server

If you are a fan of using MCP servers then you should definitely try the official VictoriaMetrics MCP Server, which enables AI-driven interaction with observability data. This MCP server allows you to use almost all read-only APIs of VictoriaMetrics, basically all the functions available in VMUI:

Querying metrics and exploring data (even drawing graphs if your client supports it)
Listing and exporting available metrics, labels, labels values and entire series
Analyzing and testing your alerting and recording rules and alerts
Showing parameters of your VictoriaMetrics instance
Exploring cardinality of your data and metrics usage statistics
Analyzing, tracing, prettifying and explaining your queries
Debugging your relabeling rules, downsampling and retention policy configurations
Integration with VictoriaMetrics Cloud

This makes it possible for MCP-compatible tools (such as Cursor, Claude Desktop, or VS Code integrations) to query and reason about observability data conversationally.

GitHub repository:
https://github.com/VictoriaMetrics-Community/mcp-victoriametrics

Video how to use VictoriaMetrics MCP server: https://www.youtube.com/watch?v=1k7xgbRi1k0

Run Your Own Local Playgrounds

In addition to these hosted playgrounds, VictoriaMetrics provides Docker and Helm-based resources you can run locally and start playing with the ecosystem:
https://hub.docker.com/u/victoriametrics
https://github.com/VictoriaMetrics/helm-charts
https://docs.victoriametrics.com/victoriametrics/quick-start/

Let me know which one was your favorite playground or if you have any feedback for improvement, ping me a comment here or you can find me on LinkedIn or GitHub.

The Unofficial Guide to Contributing to OpenTelemetry — where to look and who to talk to!

didiViking — Wed, 17 Dec 2025 09:51:50 +0000

OpenTelemetry provides the tools and standards to collect metrics, logs, and traces from applications and services. Getting started with contributions can feel overwhelming, so here is something people rarely tell you.

What People Don't Tell You

Jumping into OpenTelemetry is not just about learning the code, it is about understanding the community and the workflows behind it. Most guides focus on good first issues or setting up your local environment, but here is what you will not hear as often.
Before checking out just OpenTelemetry look at the broader ecosystem. What other tools and observability projects are out there? Are they outdated or not? Can you start contributing there as well? How do they relate to OpenTelemetry? A little bit of research and history check never hurt anyone.

Most devs don't know about this website, CLOTributor, that helps you check out good first issues from a number of Cloud Native projects. I love it!

Tips for beginners!

Start small. Contributing does not mean implementing core protocols on day one. Documentation improvements, examples, test fixes, and localization are all meaningful contributions. The codebase moves fast. APIs evolve, modules get refactored, and Collector pipelines change. Do not get discouraged if something you learned last week behaves differently today.

Community context matters. A pull request is not just code, it is discussions, design decisions and feedback from maintainers. Reading issues and pull request threads is as educational as coding itself. Your experience is your superpower. Coming from Software Engineering or developer experience gives you a fresh perspective most contributors lack. You know what patterns are practical, what documentation is confusing and which examples actually help developers adopt OpenTelemetry.

Who to Talk To

When contributing to OpenTelemetry, engage with everyone in the community, but pay particular attention to maintainers, SIG members, senior contributors and approvers. These are the people who review pull requests, set priorities and guide the direction of projects. Observing their discussions, asking thoughtful questions and seeking feedback will accelerate your learning and help your contributions align with real community needs.

Where you find them?

Useful CNCF Slack OpenTelemetry channels for new contributors:
#otel-sig-end-user, #otel-devex, #opentelemetry-new-contributors, #otel-contributor-experience, #otel-docs-localization

1. Understand the Pieces

OpenTelemetry is more than a single library, it is a set of tools and standards. SDKs exist in multiple languages including Python, Java, and Go. The Collector gathers and forwards telemetry data, and instrumentation libraries connect your frameworks and protocols. Core protocols such as OTLP, gRPC, and HTTP define how telemetry moves across systems.

If you want to go for something new and shiny in 2025, I recommend you two newer projects that are worth keeping an eye on.
OTel Injector helps automatically instrument cloud-native applications, making it easier to collect telemetry without manually modifying code.

OTel Weaver simplifies configuring and managing telemetry pipelines, reducing friction for teams setting up observability stacks.
You don't know what new and hot? Not a problem, I recommend you check out: https://opentelemetry.io/blog/2025/

Contributing to these projects allows you to influence how developers adopt OpenTelemetry in real-world environments, which is particularly valuable for SREs and platform engineers who care about scalable and maintainable monitoring. We need your developer feedback!

Direct Interviews/Feedback Sessions: The End User SIG actively seeks users for interviews to improve the project. You can reach out to them on the #otel-sig-end-user Slack channel to schedule a session.

2. Official Documentation: A Good Starting Point

The official documentation is solid but can be overwhelming if you do not know where to start. The OpenTelemetry site has quickstart guides per language, Collector setup instructions and SDK and instrumentation references. The GitHub repositories for the Collector and the Python SDK, for example, contain READMEs, contributing guides and issues labeled good first issue, which provide context and entry points for new contributors.

3. Useful Unofficial Resources: the unconventional way

Blogs, tutorials, YouTube talks, and conference sessions (e.g. KubeCon past sessions on CNCF YouTube channel) demonstrate real setups, including projects like OTel Injector. Community discussions on CNCF Slack channels often provide practical tips missing from the documentation. Contributions do not have to be code-related.
Localization of documentation, writing guides, creating sample applications, and providing feedback on instrumentation user experience are all highly valuable.

4. Setting Up a Local Sandbox

Hands-on exploration is the fastest way to learn. Start by forking and cloning the repository.

git clone https://github.com/your-username/opentelemetry-python.git cd opentelemetry-python

Set up the development environment with pip install editable for Python or go build for Go. Run tests to verify your setup.

pytest

Experiment by adding a log, tweaking instrumentation or sending data to a local Grafana instance or your preferred observability frontend platform.

5. Learn by Reading and Modifying Code

Start with small tasks such as documentation or test fixes. Study recent pull requests to learn patterns and best practices. Explore test cases as they often illustrate intended behavior clearly. Open draft pull requests for minor improvements, even for localization or example tweaks.
Be curious. Ask questions. Keep yourself motivated and challenged.

6. Expanding Your Knowledge

CNCF learning resources and courses are helpful. The Linux Foundation OpenTelemetry Certification provides a structured way to validate your understanding and gain deeper insight into the ecosystem. Hands-on projects, such as building a Collector with a sample application and Grafana pipeline, consolidate learning. Curated GitHub tutorials or Medium guides help fill gaps.

7. Making Contributions Sustainable

Sustaining contributions is about picking a focus and staying engaged. Choose a module, library, or project to focus on. Projects like OTel Injector and Weaver are particularly exciting because they address automation and pipeline management, which are pain points for many teams.

Attend SIG meetings to understand ongoing development and decisions. Share what you learn through blogs, demos, or internal talks, and review others' pull requests or mentor newcomers. Contributions that improve usability, documentation, localization, or developer experience have a long-lasting impact even if they are not code-heavy. Community is key.

Why you should listen to me?

In 2024, I was just like you. Novice, no prior Open Source or Cloud Native contribution. Then I got curious about OpenTelemetry. I joined the community and started adding my first contributions to the OpenTelemetry (OTCA) exam certification. Over the next 7–8 months, I have contributed and revised many exam questions, read official and unofficial documentation, paid attention to what my peers suggested. I showed up and I got myself motivated. With its general public release in January 2025, I became OTCA Exam developer.

Then I POC-ed OpenTelemetry at my company. It failed to reach production. But I did collaborated with developers, we instrumented code together, we made our tests, we saw the OTel potential. We provided our developer feedback to the community.

In January 2025, I contributed with my first commit and PR to the OpenTelemetry repository. Then soon followed other PRs and localization contributions. In August 2025, I created a new localization SIG for Romanian. I asked new contributors to join and mentored them.

In November 2025, at KubeCon Atlanta, the community supported us just like we supported the community all this time. Along with some of my peers, I received the OpenTelemetry 2025 Contributor Award.
In 2025, I spoke at 9 different conferences about my journey with OpenTelemetry and how important motivation is for the long run contributors in cloud native.

You can do it too. The OpenTelemetry community is extremely supportive and communicative. Be curious, ask questions, show up, diversify your contributions, stay motivated. The results won't fail to appear!

Where to find me: https://github.com/didiViking
Ask me questions: https://www.linkedin.com/in/diana-todea-b2a79968/

Article originally published on Medium, read here.