Skip the Cortex tenant headers - Deploy multi-tenant Prometheus in 5 minutes

OpenLBAC — Wed, 15 Apr 2026 20:52:49 +0000

If you've tried Cortex or Mimir for multi-tenant Prometheus, you've hit the same wall: every client needs tenant headers. Your existing Grafana dashboards break. CLI tools need updates. API integrations require modification. Your proof-of-concept becomes a migration project.

There's a better approach: query-level access control with zero client changes.

The tenant header problem

Current multi-tenant solutions require the X-Scope-OrgID header on every request:

# Every client needs modification
curl -H "X-Scope-OrgID: team-alpha" http://prometheus:9090/api/v1/query?query=up

# Grafana datasources need tenant configuration
# CLI tools need wrapper scripts
# API clients need header injection

This breaks existing infrastructure and creates security vulnerabilities - headers can be spoofed, misconfigured, or simply forgotten.

Prerequisites

Before deploying OpenLBAC, ensure you have:

Docker and Docker Compose installed
An existing Prometheus instance (or willingness to deploy one)
An OIDC provider with group claims (Keycloak, Okta, Auth0)
Basic understanding of PromQL and observability concepts

Setup: Complete multi-tenant stack

Clone and deploy the full OpenLBAC stack with example Prometheus and Keycloak:

git clone https://github.com/openlbac/openlbac
cd openlbac

# Deploy complete stack: OpenLBAC + Prometheus + Keycloak
docker-compose up -d

This starts four OpenLBAC components:

# docker-compose.yml (relevant sections)
services:
  lbac-server:
    image: openlbac/lbac-server:latest
    ports:
      - "8090:8090"
    environment:
      - DATABASE_URL=postgresql://postgres:password@postgres:5432/openlbac

  lbac-proxy:
    image: openlbac/lbac-proxy:latest
    ports:
      - "8080:8080"
    environment:
      - LBAC_CORE_URL=http://lbac-core:9090
      - UPSTREAM_PROMETHEUS_URL=http://prometheus:9090

  lbac-core:
    image: openlbac/lbac-core:latest
    ports:
      - "9090:9090"
    environment:
      - LBAC_SERVER_URL=http://lbac-server:8090

  prometheus:
    image: prom/prometheus:latest
    ports:
      - "9091:9090"
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml

Configuration: OIDC integration

Step 1: Configure OIDC provider

For Keycloak (included in docker-compose):

# Access Keycloak admin console
open http://localhost:8081

# Default credentials: admin/admin
# Create realm: "observability"
# Create groups: "platform-team", "backend-team", "frontend-team"
# Create users and assign group memberships

Step 2: Configure OpenLBAC OIDC

Update the OIDC configuration in config/oidc.yml:

oidc:
  provider_url: "http://localhost:8081/realms/observability"
  client_id: "openlbac-client"
  client_secret: "your-client-secret"
  scopes: ["openid", "profile", "groups"]
  groups_claim: "groups"

policies:
  - name: "platform-team"
    groups: ["platform-team"]
    rules:
      - label: "namespace"
        operator: "=~"
        values: ["production|staging|development"]

  - name: "backend-team"
    groups: ["backend-team"]
    rules:
      - label: "namespace"
        operator: "="
        values: ["production"]
      - label: "service"
        operator: "=~"
        values: ["api|database|cache"]

  - name: "frontend-team" 
    groups: ["frontend-team"]
    rules:
      - label: "namespace"
        operator: "="
        values: ["production"]
      - label: "service"
        operator: "="
        values: ["web"]

Testing: Query rewriting in action

Step 1: Verify policy enforcement

Test different user contexts to see query rewriting:

# Platform team member (full access)
curl -H "Authorization: Bearer $PLATFORM_TOKEN" \
  "http://localhost:8080/api/v1/query?query=up"

# Backend team member (service-filtered)
curl -H "Authorization: Bearer $BACKEND_TOKEN" \
  "http://localhost:8080/api/v1/query?query=up"

# Frontend team member (web service only)
curl -H "Authorization: Bearer $FRONTEND_TOKEN" \
  "http://localhost:8080/api/v1/query?query=up"

Step 2: Observe automatic query rewriting

Original query:

rate(http_requests_total{status="200"}[5m])

Rewritten for backend-team:

rate(http_requests_total{status="200", namespace="production", service=~"api|database|cache"}[5m])

Rewritten for frontend-team:

rate(http_requests_total{status="200", namespace="production", service="web"}[5m])

Step 3: Test with existing tools

Point your existing tools to OpenLBAC instead of Prometheus directly:

# Grafana datasource configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  prometheus.yml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://lbac-proxy:8080  # Changed from prometheus:9090
        access: proxy
        jsonData:
          httpHeaderName1: "Authorization"
        secureJsonData:
          httpHeaderValue1: "Bearer $GRAFANA_OIDC_TOKEN"

Your existing dashboards work unchanged. CLI tools work unchanged. The only change is the endpoint URL.

Optimization: Production deployment

Horizontal scaling

Scale proxy instances for high throughput:

# docker-compose.override.yml
services:
  lbac-proxy:
    deploy:
      replicas: 3

  nginx:
    image: nginx:alpine
    ports:
      - "8080:80"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf

Nginx configuration for load balancing:

upstream lbac_proxies {
    server lbac-proxy-1:8080;
    server lbac-proxy-2:8080;
    server lbac-proxy-3:8080;
}

server {
    listen 80;
    location / {
        proxy_pass http://lbac_proxies;
        proxy_set_header Authorization $http_authorization;
    }
}

Monitoring and alerting

Monitor OpenLBAC components:

# prometheus.yml
global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'lbac-proxy'
    static_configs:
      - targets: ['lbac-proxy:8080']
    metrics_path: '/metrics'

  - job_name: 'lbac-core'
    static_configs:
      - targets: ['lbac-core:9090']
    metrics_path: '/metrics'

Key metrics to monitor:

lbac_proxy_query_duration_seconds - Query rewriting latency
lbac_proxy_queries_total - Request rate and error rate
lbac_core_policy_updates_total - Policy propagation health
lbac_core_connected_proxies - Proxy connectivity status

Audit and compliance

Enable audit logging for compliance frameworks:

# config/audit.yml
audit:
  enabled: true
  sink_type: "elasticsearch"
  elasticsearch:
    endpoint: "http://elasticsearch:9200"
    index: "openlbac-audit"

  events:
    - "query_execution"
    - "policy_violation"
    - "authentication_failure"
    - "authorization_decision"

Each query execution logs:

User identity and group membership
Original query and rewritten query
Data sources accessed
Timestamp and request metadata

Troubleshooting common issues

Issue: JWT token validation fails

Symptoms: 401 Unauthorized responses from lbac-proxy

Solution: Verify OIDC configuration and token format:

# Check token structure
echo $TOKEN | base64 -d | jq .

# Verify groups claim exists
# Ensure issuer matches provider_url
# Check client_id in token audience

Issue: Queries return empty results

Symptoms: Valid queries return no data after policy application

Solution: Check policy rule logic and label presence:

# Verify labels exist in your metrics
curl http://prometheus:9090/api/v1/labels

# Test policy rules manually
curl "http://localhost:8090/api/v1/policies/test" \
  -d '{"query": "up", "user_groups": ["backend-team"]}'

Issue: High query latency

Symptoms: Increased response times after OpenLBAC deployment

Solution: Optimize policy complexity and caching:

# config/performance.yml
cache:
  policy_cache_ttl: 300s
  query_cache_ttl: 60s

optimization:
  max_rule_complexity: 10
  parallel_policy_evaluation: true

OpenLBAC adds <5ms latency in most configurations. Higher latencies indicate policy complexity issues or network bottlenecks.

Why this approach works better

Zero client modification: Existing tools continue working unchanged. No headers to manage or spoof.

Real-time enforcement: Policies apply at query execution, not just dashboard level. API access and CLI tools automatically controlled.

Enterprise IdP integration: Leverage existing Keycloak, Okta, or Auth0 groups. No custom authentication development required.

Comprehensive audit: Every query decision logged for PCI DSS, GDPR, and SOC 2 compliance requirements.

Five-minute deployment: Docker Compose to production-ready in minutes, not weeks of client migration.

The tenant header approach assumes you can modify every client. The query rewriting approach assumes you can't - and works anyway.

What's your experience with multi-tenant observability? Have you hit the tenant header wall, or found other approaches that preserve existing integrations?

DEV Community: OpenLBAC