DEV Community: Giam

Logs, Metrics, Traces… Leaks? The Case for Auditable Observability

Giam — Wed, 21 May 2025 09:45:00 +0000

You’ve got dashboards. You’ve got alerts. You’ve got logs going back six months. But can you answer this one question?

“Who accessed this data, and were they supposed to?”

Welcome to the next era of observability: auditable observability — where what you collect isn’t the only concern. It’s about who sees it, how it’s protected, and what trails they leave behind.

And if you think your observability stack is ready, think again.

The Ugly Truth About Modern Observability

Observability has evolved fast. In every SRE and DevOps team I know, there’s a rich landscape of:

Prometheus for metrics ✅
Loki or Elasticsearch for logs ✅
Tempo or Jaeger for traces ✅
Grafana dashboards on top ✅

What’s missing?
Access control. Audit trails. Data governance.

Here’s what that looks like in real life:

A junior engineer accidentally sees production logs from a different tenant.
An on-call rotation includes people who can query everything.
Nobody knows who queried sensitive traces — because there’s no log of queries at all.
An auditor asks you for proof that no one tampered with historical metrics… and you come up blank.

Observability without auditability is a security risk in disguise.

The Cost of Getting It Wrong

Regulations like SOX, HIPAA, and GDPR aren’t just legal footnotes — they’re expectations:

Immutable audit logs must be stored securely.
Access to observability data must be logged and governed.
Personal data in logs must be protected, redacted, or deleted on request.

If your observability stack can’t answer:

“Who accessed logs from this time window?”
“Can we prove this trace wasn’t altered?”
“Which team owns this metric?”

…then you’re flying blind in the eyes of auditors, security teams, and your own customers.

Observability Was Built for Visibility — Not Auditability

Most open-source tools weren’t built with multi-tenancy or compliance in mind.

Take Prometheus, Loki, or Tempo:

They’re great at ingesting data.
They’re okay at organizing it by labels.
They’re terrible at controlling who can access what.

RBAC in Grafana helps — but only at the dashboard level.
It doesn’t restrict queries. It doesn’t filter log lines. It doesn’t know about labels that matter to your platform (e.g. tenant_id, team, region, sensitivity=high).

So most teams try to patch it:

Build reverse proxies that inject tenant labels.
Run a Grafana per customer or per team.
Hope everyone plays nice with naming conventions.

It’s fragile. It’s hard to maintain. And it scales poorly.

The Shift Toward Auditable Observability

In 2025, the trend is clear:

Observability is no longer just about uptime.
It’s part of your security surface.
It must be auditable, accountable, and access-controlled.

Leading teams are asking:

“Can we enforce RBAC or LBAC on observability data itself?”
“Do our logs have traceability and tamper-resistance?”
“Are our metrics labeled in a way that enables per-tenant access control?”

These aren’t nice-to-haves anymore. They’re table stakes — especially for:

Multi-tenant platforms
Enterprise customers
Healthcare, fintech, and regulated industries
Security-conscious orgs under zero-trust mandates

What Auditable Observability Looks Like

Here’s what good looks like in 2025:

✅ Logs, metrics, and traces are labeled by tenant, team, and sensitivity
✅ Access to data is scoped by labels, roles, and tokens
✅ All queries are logged, audit trails are immutable
✅ Data is stored in long-term object storage with WORM (Write Once, Read Many) policies
✅ Observability is integrated into compliance reports and security reviews

How We Solved It

This problem frustrated us for way too long.

We had all the right tools — but no way to control or audit observability data access at the level we needed.

So we built usegiam.com — a lightweight access control platform that brings observability governance to your stack.

🔐 What it does:

Enforce who can access what, based on labels in Prometheus, Loki, Tempo, and Thanos and more
Add query filters automatically based on the user/team/service
Audit all queries - with tamper-resistant trails
Integrate with your SSO and RBAC provider
Deploy as a proxy in front of your observability backends — no vendor lock-in

Built for SREs. Loved by compliance teams.
No need to go Enterprise just to keep your data safe.

Why Now?

You wouldn’t ship code without access control.
You wouldn’t expose S3 buckets without IAM.

So why are we still exposing logs and metrics to everyone?

With the rise of platform engineering, multi-tenant environments, and cloud-native complexity, observability is security-critical infrastructure.

And the teams that realize this first?
They’re going to be the ones who sleep best during the next incident — or audit.

TL;DR

Observability data is sensitive — and often unprotected.
Most stacks don’t enforce proper access control or audit trails.
Regulations require logs and metrics to be immutable, scoped, and monitored.
You need observability that’s not just visible — but auditable.

👉 If this hits close to home, try usegiam.com.
Start making your observability stack secure, compliant, and access-aware.

Are Your Loki Logs Safe? Scalable Solutions for Segregation and Authorization

Giam — Sun, 11 May 2025 15:28:53 +0000

1. Introduction: Logs, Scale, and Responsibility

In today’s cloud-native world, logs are more than just developer breadcrumbs; they are operational lifelines, forensic goldmines, and, in regulated environments, potential liabilities. As engineering teams scale, so do their logs — and with scale comes complexity: multiple teams, multiple environments, and often, multiple tenants.

Grafana Loki has rapidly emerged as a go-to log aggregation system thanks to its horizontal scalability, and seamless integration with the wider Grafana stack. But as more teams adopt Loki to centralize logging, a crucial question arises: who should have access to which logs? Centralization without control invites chaos — which is why forward-thinking teams are looking beyond the basics.

While Loki shines in many areas, log segregation and access control require careful planning, execution and maintenance. This guide dives deep into how Loki handles log isolation, authentication, and authorization — and how to scale it securely and sanely.

2. Understanding Loki’s Architecture and Log Model

At its core, Loki is designed for efficiency. Unlike systems like Elasticsearch or Splunk, Loki doesn’t index the full content of logs. Instead, it indexes only metadata: the labels. These are key-value pairs like app, namespace, or env, attached to each log stream at ingestion.

Loki is horizontally scalable and built for cloud-native environments. Logs are pushed to Loki by agents such as Promtail, Fluent Bit, or the Grafana Agent. Loki chunks, compresses, and stores log data in an object store (e.g., S3, GCS), while only maintaining a lightweight index of labels.

This architecture drastically reduces operational cost and complexity but comes with trade-offs. This means that access control with Loki is not based on user identity or log contents, but on metadata boundaries.

3. What Is Log Segregation and Why It Matters

Log segregation is the practice of logically and securely separating log data based on its origin, sensitivity, or intended audience. Whether you're managing logs for hundreds of microservices, multiple teams, or even customer tenants, the ability to partition logs is critical.

Why it matters:

Security: Prevent unauthorized access to sensitive data (e.g., customer PII, internal system details).
Compliance: Meet regulations like GDPR, HIPAA, SOC2 which require data access controls and auditability.
Clarity: Reduce cognitive noise by showing engineers only the logs relevant to their service or team.

Without proper segregation, logs from staging and production, or across teams, can easily be commingled — leading to confusion, accidental data leaks, or worse.

4. Multi-Tenancy in Loki: The First Line of Isolation

Loki supports multi-tenancy via the X-Scope-OrgID HTTP header. When this is enabled (via auth_enabled: true), every log push or query must include this header to identify the tenant.

This provides hard isolation:

Each tenant’s logs are stored and indexed separately.
Queries scoped to one tenant cannot access another tenant’s logs.

Common ways to map tenants:

One tenant per team (e.g., team-payments)
One tenant per environment (e.g., prod, dev)
One tenant per customer (for SaaS platforms)

While effective, managing large numbers of tenants can become operationally heavy. Loki has no native UI or API for tenant management in OSS. At scale, you need to track tenant IDs, assign them to agents, enforce limits, and ensure proper access from front-ends like Grafana.

In fact, Grafana's enterprise offering is often required to manage multi-tenancy cleanly - especially if you're looking to apply data source permissions, access policies, or provide per-tenant visibility through Grafana itself. That can quickly become costly and complex to manage, particularly for teams operating in hybrid or self-hosted environments.

This is where lightweight, purpose-built access management solutions like Giam shine. GIAM provides a simple but powerful way to enforce and manage tenant boundaries on top of Loki — without requiring a full Grafana Enterprise stack. It gives teams a centralized UI and policy engine to define which users or services can access what, across tenants and environments.

5. Labels: Useful for Querying, Not for Securing

Labels are essential in Loki for organizing and querying logs efficiently. But labels are not a security mechanism.

Why?

Any user with access to a tenant can query any label combination.
There is no built-in way to restrict access to certain labels or streams.

It's a common trap: relying on labels like env="prod" or team="A" for access control. But unless you pair this with enforced tenant separation, anyone with access to that tenant can still query {env="prod"}.

The best practice: use labels for structure and hygiene, but enforce true security boundaries with tenants.

That said, many real-world setups involve shared tenants or overlapping responsibilities - especially during migrations, shared infrastructure operations, or multi-team debugging. In such cases, organizations need fine-grained, label-aware authorization on top of tenant boundaries.

This is where Giam becomes especially powerful: it allows teams to define access policies not just at the tenant level, but at the label or log stream level. For example, Giam can enforce that a user can only query logs where team="frontend" or env="dev", even within a shared tenant - enabling safer collaboration without over-segmentation or extra Loki clusters.

6. Authentication & Authorization in Loki: What’s Built In (and What’s Not)

By default, open-source Loki provides no authentication or authorization. It assumes you're running it behind an external proxy, gateway, or identity provider.

Common setups include:

OAuth2-Proxy in front of Loki, authenticating users and injecting the correct X-Scope-OrgID.
NGINX with Basic Auth for basic protection.
kube-rbac-proxy to integrate with Kubernetes RBAC.
Grafana as a front-end, using per-org configurations and multiple data sources.

However, this approach pushes a lot of complexity to the edge:

You must correctly map users to tenants.
You must ensure headers can't be spoofed.
There's no built-in concept of users, roles, or audit logs.

While feasible, this creates a fragile setup. As organizations grow, the need for robust, centralized access control becomes clear.

Giam steps in here as a purpose-built solution for this very gap. Rather than duct-taping identity and policy enforcement across proxies, Giam acts as a secure, centralized guardrail for Loki — validating user identity, injecting tenant context, and applying access rules consistently. It reduces the overhead of maintaining proxy chains and enables structured policy enforcement that grows with your team.

7. Real-World Challenges: From Compliance to Complexity

As teams and environments grow, the cracks start to show:

Tenant sprawl: Without lifecycle management, tenant IDs accumulate. Misconfiguration leads to data going to the wrong place.
Privilege creep: Shared access tokens or data sources give broad access over time.
Audit gaps: With no query-level logging in OSS, it's hard to trace who accessed what.
Policy enforcement: There's no way to enforce “User A can only query logs from app X in environment Y”.

In regulated industries, this becomes untenable. For example, a healthcare company must ensure that engineers can only access logs for services they're responsible for, and that all access is auditable.

Giam address these growing pains by offering a centralized control plane for log access — enabling policy enforcement, per-user audit trails, and scalable access governance without stitching together multiple tools.

8. Best Practices for Scalable and Secure Log Segregation

To operate Loki securely at scale:

Enable multi-tenancy early. Don’t delay separating teams or environments.
Use infrastructure as code to manage tenants and access. Helm charts and Terraform modules help.
Guard Loki with authentication: use OAuth2-Proxy or similar to verify identity and inject tenant headers.
Separate ingestion and query endpoints: often different services need different access levels.
Implement query auditing at the proxy or Grafana level.
Track per-tenant usage and errors using Loki’s own metrics.
Alert on anomalies: unusual query patterns, volume spikes, or failed auth.

These steps help minimize blast radius, improve accountability, and ease operations — and with tools like usegiam.com, many of these practices can be automated and standardized, making them less error-prone and more consistent across teams.

9. The Case for Advanced Access Control

There comes a point where reverse proxies, scripts, and convention-based policies start to buckle. At this stage, many teams turn to centralized solutions like usegiam.com that bring policy-driven access control, auditability, and seamless IAM integration — without the usual complexity. Large organizations need:

Centralized tenant management
Fine-grained RBAC/ABAC policies
API tokens with scoped permissions
Audit logs for queries and access events
Seamless integration with enterprise SSO/IAM

These aren't luxuries — they're requirements for secure, compliant observability. Solutions that offer policy-driven log access control, visibility into usage, and delegation of access management to teams can be a game-changer.

They reduce operational burden, prevent misconfigurations, and provide confidence that your log data isn’t becoming a liability.

10. Conclusion: Making Loki Work for You — Securely and Sanely

Loki lays a rock-solid foundation for log aggregation — it’s scalable, cost-effective, and fits naturally into the Grafana ecosystem. But when it comes to access control, segregation, and security, it deliberately keeps things light, leaving much of the responsibility to you.

For smaller teams, that simplicity can be an advantage. A few proxies, some label discipline, and you’re up and running. But as platforms grow, so do the stakes — and so does the need for real access governance.

At scale, you need more than good intentions. You need policy enforcement, auditability, and fine-grained access, all without slowing down your developers. That’s where purpose-built solutions come into play.

With the right architecture, Loki becomes more than a place to stash logs — it becomes a secure, self-service platform for every team. Platforms like usegiam.com help you make that leap — delivering the controls, guardrails, and visibility you need to operate responsibly at scale, without rebuilding everything from scratch.

RBAC vs. LBAC: Which Scales for Multi-Tenant Dashboards?

Giam — Mon, 05 May 2025 08:00:00 +0000

The Multi-Tenant Dashboard Dilemma
Multi-tenancy is a reality for many DevOps and platform teams, especially when managing observability platforms like Grafana. Ensuring secure, isolated data access for multiple tenants often reveals limitations in traditional RBAC solutions, prompting a shift toward more granular methods like LBAC.

This article explores RBAC and LBAC in Grafana, highlighting real-world best practices, pitfalls to avoid, and actionable insights to scale efficiently.

RBAC in Grafana: The Basics

Role-Based Access Control (RBAC) assigns permissions based on defined roles (Admin, Editor, Viewer), controlling user actions like dashboard creation or data source management. RBAC excels in simplicity, making it ideal for straightforward access needs in small, single-tenant setups.

Pros:

Easy to implement
Well-understood by engineers Cons:
Quickly becomes unmanageable at scale
Limited granularity, causing "role explosion" in complex scenarios

What is LBAC (Label-Based Access Control)?

LBAC enhances access control by dynamically filtering data based on labels or attributes at query-time, rather than just static user roles. Grafana recently introduced LBAC for its enterprise data sources, enabling teams to share dashboards safely across tenants by enforcing label-specific policies.

Pros:

Fine-grained data isolation
Scalable across multiple tenants without duplicating dashboards or data sources

Cons:

Initial setup complexity

Feature	RBAC	LBAC
Granularity	Coarse (role-based)	Fine (label-based)
Management Ease	Easier initially; complexity grows	Complex initially; scalable long-term
Scalability	Limited due to role explosion	Highly scalable via dynamic policies
Data Isolation	Limited data-level control	Strong data-level isolation

RBAC alone can quickly lead to unwieldy role management. LBAC complements RBAC by handling data-level permissions dynamically, ensuring each tenant sees only relevant data.

Challenges in Multi-Tenant Grafana with RBAC Only

Teams attempting multi-tenancy with only RBAC encounter significant challenges:

Role Explosion: Creating numerous roles per tenant quickly becomes unmanageable.
Security Risks: Shared data sources may inadvertently expose data across tenants due to insufficient query-level restrictions.
Operational Overhead: Separate Grafana instances or duplicated dashboards are resource-intensive and inefficient.

How LBAC Solves These Challenges

Label-based policies address these challenges head-on:

Simplified Management: One shared Grafana instance with LBAC reduces overhead.
Enhanced Security: Dynamic, per-request data filtering ensures tenants only see permitted data.
Greater Flexibility: Easily onboard new tenants or update policies without structural changes.

RBAC and LBAC Together: Best Practices for Multi-Tenancy

For multi-tenant dashboards, relying solely on RBAC can become complex and ineffective. Combining RBAC with LBAC is a recommended best practice:

Step 1: Establish RBAC Roles and Teams

Start by clearly defining roles such as Admin, Editor, and Viewer. Assign these roles based on specific responsibilities—like dashboard creation, editing, or read-only access—adhering strictly to the principle of least privilege. Group users into teams when possible, regularly reviewing permissions to maintain appropriate access levels.

Step 2: Introduce Label-Based Policies

Enhance security by applying LBAC to dynamically filter data visibility at query-time, using labels like tenant IDs, environment names, customer segments, or project identifiers. Properly labeled metrics and logs ensure that each tenant sees only the data they are authorized to access.

For instance, data sources might include labels such as tenant_id: "tenant123" or environment: "production". Queries automatically enforce these label constraints, providing strict data isolation without manual intervention.

This model scales especially well in shared environments. For instance, a managed service provider running a single Grafana instance can safely serve multiple clients by filtering queries on tenant labels, ensuring strict data isolation without added operational burden.

Step 3: Choose the Right Tool for Enforcement

Grafana Cloud and Grafana Enterprise both offer native RBAC and LBAC features, which are excellent options for organizations already invested in Grafana’s commercial offerings. However, these solutions can quickly become costly, especially for SMBs, smaller teams, or users of Grafana Open Source, which lacks built-in LBAC capabilities.

Tools like Giam offer an efficient alternative. Giam integrates seamlessly with Grafana, providing external enforcement of both RBAC and LBAC without plugins, custom forks, or code changes. Policies are managed centrally and enforced per-request, providing precise and scalable access control.

Benefits of Combining RBAC and LBAC with Giam:

Centralized Management: All policies managed from a single, intuitive interface.
Granular Data Isolation: Tenant-specific data is protected at the query-level, eliminating risk of accidental exposure.
Audit Trail: Detailed logs of data access ensure compliance with security audits.
Zero Operational Overhead: No separate Grafana instances or duplicate dashboards required—significantly reducing administrative burden.

Looking Ahead: Future Trends

As "zero trust" security gains traction, dynamic access control methods like LBAC are increasingly critical. Expect continuous evolution toward hybrid RBAC+LBAC models, supported by external policy enforcement tools, simplifying secure multi-tenancy at scale.

Conclusion

Which scales better, RBAC or LBAC? The answer: both, used together. RBAC lays a manageable foundation, while LBAC provides granular control needed for secure, scalable multi-tenancy. Solutions like Giam offer seamless integration, avoiding the complexity of custom solutions and empowering DevOps teams to focus on delivering value.

Curious to see RBAC + LBAC in action for your dashboards? Request a demo and experience simpler, more secure multi-tenant dashboards today.