When we write code, it often feels like the main thing is to make it work locally. But reality is different: the real life of a service begins not on your laptop, but in production. That’s where it faces load, unpredictable users, and dozens of unexpected situations.
Many developers think of monitoring and logs as “something for DevOps.” Let them look at the graphs and figure out why the service crashed. But the truth is: observability is a developer’s tool first. It helps you quickly understand what went wrong and fix a bug before it turns into a midnight call from support.
In this post I’ll explain:
- 🧐 why metrics and dashboards are important,
- 🔧 how developers can actually use them,
- 🚨 and why without observability you’re basically “flying blind” in production.
What Is Observability (in simple words)
Any service lives in two worlds:
- 🌍 the code world, where you’re in control with
printor breakpoints, - 🌍 and the production world, where the system runs under load, with real users and unpredictable inputs.
Observability is how you understand what’s happening inside without looking directly into the code.
The three “pillars” are:
- 📜 Logs — text outputs (errors, requests, events). Answer: “what happened?”
- 📊 Metrics — numbers (request counts, response times, memory usage). Answer: “how is it working?”
- 🧵 Traces — following one request across multiple services. Answer: “why is it working this way?”
Together, they give you a complete picture: find bottlenecks, debug errors, and see how changes affect production.
The Main Tools (and Why These)
There are many monitoring products out there — from Datadog and New Relic to Dynatrace. In practice, most projects I see use the Grafana Labs + Prometheus ecosystem.
-
⏱ Prometheus
- The de-facto standard for metrics collection.
- Simple model: the service exposes metrics via HTTP, Prometheus “scrapes” them.
- Powerful query language (PromQL).
- Tons of exporters: MySQL, PostgreSQL, Nginx, Docker, etc.
-
📈 Grafana
- Universal visualization tool.
- Works not only with Prometheus but also with Loki, Elastic, Postgres, and more.
- Great dashboards, alerts, annotations.
- Huge marketplace of ready-to-use dashboards.
-
📜 Loki
- Centralized logging system with Prometheus-like syntax.
- Much lighter than Elasticsearch.
- Perfectly integrated with Grafana — jump from a metric to the exact log line.
-
🧵 Tempo / Jaeger
- Distributed tracing.
- Shows the full path of a request across multiple services — essential for microservices.
Why developers love this stack:
✔️ open source — no license cost, easy to start
✔️ low entry barrier — a basic dashboard in a couple of hours
✔️ one ecosystem — everything integrates
✔️ scalable — from pet projects to highload
How Developers Can Actually Use It
Here are some everyday scenarios:
🔎 1. Found a bug you can’t reproduce
- User says: “The button doesn’t work.”
- Search in Loki by
request_idor exception text. - You see the stacktrace → problem identified. 👉 Result: no more “works on my machine.”
⚡ 2. Service slows down after a release
- Grafana shows latency doubled after yesterday’s deploy.
- Drill down into endpoint metrics → find the culprit.
- Check logs for parameters. 👉 Result: you know exactly what broke.
🧮 3. Optimizing SQL queries
- Add metrics for query count and duration.
- Grafana shows the heaviest queries.
- Compare before/after optimization. 👉 Result: real data, not gut feeling.
🚨 4. Reacting to alerts yourself
- Rule: “if 5xx > 5% over 5 minutes → send alert.”
- Notifications in Slack/Telegram. 👉 Result: developers react instantly, not a day later via support.
🛠️ 5. Your own “developer dashboard”
- A few panels: error rates, latency, top endpoints. 👉 Result: all essentials in one place.
A Real Project Example
I joined a growing gaming project (FastAPI + MySQL) as a part-time engineer. It was unstable — clients often complained that the game wouldn’t load.
What I found:
- ❌ no log collection,
- ❌ Grafana only had node-exporter metrics nobody watched. The project was flying blind.
Step 1. Logs
- Set up Loki with JSON format.
- Added request/response logs, duration, endpoint filtering.
- Super fast search via Docker plugin.
Step 2. Metrics
- Added HTTP latency, DB transaction times, response statuses, concurrent requests.
- Finally saw the full load picture.
Step 3. Problem discovery
- MySQL queries crashing → fixed with connection pooling.
- httpx connection pool overloading → fixed.
- Long queries → Percona Monitoring and Management → cleaned up DB load.
Step 4. Alerts
- 500 errors after release.
- RabbitMQ queue overflow.
- Network issues.
- Payment provider outages → switch to backup quickly.
The result:
- ✅ from black box → transparent system
- ✅ developers could debug themselves
- ✅ support and ops stopped firefighting
- ✅ downtime minimized → saved money
What About Tracing?
Tracing is important, especially in microservices. But for basics, I use a simpler approach: passing a request_id across all services.
That lets me track a request in Loki end-to-end across the ecosystem. It’s not as powerful as Jaeger or Tempo, but it covers 80% of a developer’s needs.
A Minimal Observability Checklist
If your project has nothing yet, start small — 20% effort for 80% results:
✅ Logs
- Centralized (Loki/EFK)
- JSON format
- Add
request_idand duration - Filter by endpoint/error level
✅ Metrics
- HTTP latency
- Response statuses (2xx/4xx/5xx)
- Concurrent requests
- DB transaction times
- Queue load
✅ Database
- MySQL/Postgres exporter
- Track slow queries, deadlocks
✅ Alerts
- 5xx error threshold
- Queue overflow
- Service outages
- DB overload
✅ Team Interface
- Base Grafana dashboard: errors, latency, load
- A “developer dashboard” with API essentials
👉 With just this, your project stops being a black box. Everything else — tracing, business dashboards, analytics — can be added gradually.
Originally published at mind.mansur.expert.
Top comments (0)