DEV Community

FABIOLA ESTEFANI POMA MACHICADO
FABIOLA ESTEFANI POMA MACHICADO

Posted on

Observability Practices in Modern Software Systems

🌐What is Observability?
Observability is the ability to infer the internal state of a system by examining its outputs—primarily through logs, metrics, and traces. It extends traditional monitoring by allowing dynamic querying and root cause analysis of system behavior.
📈 Why Observability Matters
In modern microservices and distributed systems, traditional monitoring is no longer enough. Observability helps by:

  • Detecting anomalies proactively
  • Reducing Mean Time To Detect (MTTD) and Mean Time To Resolve (MTTR)
  • Supporting real-time diagnostics
  • Enabling end-to-end visibility across components

🔧 Real-World Example: Observability with Prometheus and Grafana in Node.js
We'll monitor a simple Node.js API and visualize request metrics on Grafana.
🛠️ Setup Overview

  • Language: Node.js (Express)
  • Metric Collector: Prometheus
  • Visualization: Grafana
  • Exporter: prom-client (Node.js metrics exporter) 📦 Install Dependencies 
npm init -y
npm install express prom-client
Enter fullscreen mode Exit fullscreen mode

🧪 Sample Application (server.js)

const express = require('express');
const client = require('prom-client');

const app = express();
const collectDefaultMetrics = client.collectDefaultMetrics;
collectDefaultMetrics();

const httpRequestDurationMicroseconds = new client.Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in ms',
  labelNames: ['method', 'route', 'code'],
  buckets: [0.1, 0.3, 0.5, 1, 1.5]
});

app.use((req, res, next) => {
  const end = httpRequestDurationMicroseconds.startTimer();
  res.on('finish', () => {
    end({ method: req.method, route: req.route?.path || req.url, code: res.statusCode });
  });
  next();
});

app.get('/', (req, res) => {
  res.send('Hello, world!');
});

app.get('/metrics', async (req, res) => {
  res.set('Content-Type', client.register.contentType);
  res.end(await client.register.metrics());
});

app.listen(3000, () => {
  console.log('Server running on http://localhost:3000');
});
Enter fullscreen mode Exit fullscreen mode

📡 Prometheus Configuration (prometheus.yml)

global:
  scrape_interval: 10s

scrape_configs:
  - job_name: 'node_app'
    static_configs:
      - targets: ['localhost:3000']
Enter fullscreen mode Exit fullscreen mode

📊 Visualizing Metrics with Grafana

  • Add Prometheus as a data source in Grafana
  • Create a dashboard with a panel using this query: 
rate(http_request_duration_seconds_count[1m])
Enter fullscreen mode Exit fullscreen mode

You can visualize per-route traffic, latency buckets, or status code breakdown.

🎯 Best Practices for Observability

  • Expose /metrics endpoints for all services
  • Use structured logging with correlation IDs
  • Label metrics with meaningful tags (method, route, status)
  • Automate dashboards for services and infrastructure
  • Set alerts based on SLOs/SLAs

🧩 Wrapping Up
Observability isn't just about tools—it's a mindset. By embracing metrics, logs, and traces from the ground up, you build software that is easier to debug, scale, and maintain.
If you’d like, I could also show you how to push logs to the ELK stack or wire traces using OpenTelemetry in a similar setup. Want to take it a step further with something like MCP integration or Power BI for correlating metrics to thesis outcomes? I’m all ears.

Top comments (0)