DEV Community: Mansur Fattakhov

How Developers Can Monitor Production — and Why It Matters

Mansur Fattakhov — Sun, 13 Sep 2026 08:21:47 +0000

When we write code, it often feels like the main thing is to make it work locally. But reality is different: the real life of a service begins not on your laptop, but in production. That’s where it faces load, unpredictable users, and dozens of unexpected situations.

Many developers think of monitoring and logs as “something for DevOps.” Let them look at the graphs and figure out why the service crashed. But the truth is: observability is a developer’s tool first. It helps you quickly understand what went wrong and fix a bug before it turns into a midnight call from support.

In this post I’ll explain:

🧐 why metrics and dashboards are important,
🔧 how developers can actually use them,
🚨 and why without observability you’re basically “flying blind” in production.

What Is Observability (in simple words)

Any service lives in two worlds:

🌍 the code world, where you’re in control with print or breakpoints,
🌍 and the production world, where the system runs under load, with real users and unpredictable inputs.

Observability is how you understand what’s happening inside without looking directly into the code.

The three “pillars” are:

📜 Logs — text outputs (errors, requests, events). Answer: “what happened?”
📊 Metrics — numbers (request counts, response times, memory usage). Answer: “how is it working?”
🧵 Traces — following one request across multiple services. Answer: “why is it working this way?”

Together, they give you a complete picture: find bottlenecks, debug errors, and see how changes affect production.

The Main Tools (and Why These)

There are many monitoring products out there — from Datadog and New Relic to Dynatrace. In practice, most projects I see use the Grafana Labs + Prometheus ecosystem.

⏱ Prometheus
- The de-facto standard for metrics collection.
- Simple model: the service exposes metrics via HTTP, Prometheus “scrapes” them.
- Powerful query language (PromQL).
- Tons of exporters: MySQL, PostgreSQL, Nginx, Docker, etc.
📈 Grafana
- Universal visualization tool.
- Works not only with Prometheus but also with Loki, Elastic, Postgres, and more.
- Great dashboards, alerts, annotations.
- Huge marketplace of ready-to-use dashboards.
📜 Loki
- Centralized logging system with Prometheus-like syntax.
- Much lighter than Elasticsearch.
- Perfectly integrated with Grafana — jump from a metric to the exact log line.
🧵 Tempo / Jaeger
- Distributed tracing.
- Shows the full path of a request across multiple services — essential for microservices.

Why developers love this stack:

✔️ open source — no license cost, easy to start

✔️ low entry barrier — a basic dashboard in a couple of hours

✔️ one ecosystem — everything integrates

✔️ scalable — from pet projects to highload

How Developers Can Actually Use It

Here are some everyday scenarios:

🔎 1. Found a bug you can’t reproduce

User says: “The button doesn’t work.”
Search in Loki by request_id or exception text.
You see the stacktrace → problem identified. 👉 Result: no more “works on my machine.”

⚡ 2. Service slows down after a release

Grafana shows latency doubled after yesterday’s deploy.
Drill down into endpoint metrics → find the culprit.
Check logs for parameters. 👉 Result: you know exactly what broke.

🧮 3. Optimizing SQL queries

Add metrics for query count and duration.
Grafana shows the heaviest queries.
Compare before/after optimization. 👉 Result: real data, not gut feeling.

🚨 4. Reacting to alerts yourself

Rule: “if 5xx > 5% over 5 minutes → send alert.”
Notifications in Slack/Telegram. 👉 Result: developers react instantly, not a day later via support.

🛠️ 5. Your own “developer dashboard”

A few panels: error rates, latency, top endpoints. 👉 Result: all essentials in one place.

A Real Project Example

I joined a growing gaming project (FastAPI + MySQL) as a part-time engineer. It was unstable — clients often complained that the game wouldn’t load.

What I found:

❌ no log collection,
❌ Grafana only had node-exporter metrics nobody watched. The project was flying blind.

Step 1. Logs

Set up Loki with JSON format.
Added request/response logs, duration, endpoint filtering.
Super fast search via Docker plugin.

Step 2. Metrics

Added HTTP latency, DB transaction times, response statuses, concurrent requests.
Finally saw the full load picture.

Step 3. Problem discovery

MySQL queries crashing → fixed with connection pooling.
httpx connection pool overloading → fixed.
Long queries → Percona Monitoring and Management → cleaned up DB load.

Step 4. Alerts

500 errors after release.
RabbitMQ queue overflow.
Network issues.
Payment provider outages → switch to backup quickly.

The result:

✅ from black box → transparent system
✅ developers could debug themselves
✅ support and ops stopped firefighting
✅ downtime minimized → saved money

What About Tracing?

Tracing is important, especially in microservices. But for basics, I use a simpler approach: passing a request_id across all services.

That lets me track a request in Loki end-to-end across the ecosystem. It’s not as powerful as Jaeger or Tempo, but it covers 80% of a developer’s needs.

A Minimal Observability Checklist

If your project has nothing yet, start small — 20% effort for 80% results:

✅ Logs

Centralized (Loki/EFK)
JSON format
Add request_id and duration
Filter by endpoint/error level

✅ Metrics

HTTP latency
Response statuses (2xx/4xx/5xx)
Concurrent requests
DB transaction times
Queue load

✅ Database

MySQL/Postgres exporter
Track slow queries, deadlocks

✅ Alerts

5xx error threshold
Queue overflow
Service outages
DB overload

✅ Team Interface

Base Grafana dashboard: errors, latency, load
A “developer dashboard” with API essentials

👉 With just this, your project stops being a black box. Everything else — tracing, business dashboards, analytics — can be added gradually.

Originally published at mind.mansur.expert.

How to Bring Order to a Project: A Track Map

Mansur Fattakhov — Sun, 13 Sep 2026 08:21:43 +0000

Every now and then I join a project where "everything works, but nobody dares to touch it." Deployment is done by hand from memory, the architecture grew historically, knowledge lives in people's heads, hiring happens by accident. That's normal: this is what almost any project looks like when it grows faster than it matures.

You don't bring order to a project like that in one heroic push. What works is different: split the chaos into tracks, take small steps within each track, and make every step pay off immediately. Below is my complete plan — what to do, in what order, and why. Each track will get its own deep-dive article; this is the map.

How to use this plan

Snapshot first, then light, then changes. Don't fix anything until you can see what's going on.
Tracks run in parallel, at different speeds — push the one that hurts right now.
A step must fit into a week. If it doesn't, it's not a step, it's a project; keep cutting.
Every touch leaves a trace: touched a service — bring it closer to the standard; figured out a piece of the system — write it down.

Track 0. State snapshot

One week, fix nothing — only look and write down.

⬜ Service map: what exists, where it lives, who deploys it and how

⬜ The "scary places" list — things people describe as "better not touch it"

⬜ Who knows what: which knowledge has a single carrier

⬜ Top 5 risks — one page, in plain human language

💬 The main artifact of this stage is not a document but a picture in your head. The document is a way to verify it: show it to the team, let them correct you.

📖 Full article: the state snapshot — your first week on a new project.

Track 1. Observability — turn on the lights

⬜ Centralized, structured logs

⬜ HTTP and database metrics

⬜ 3–5 alerts for the things that actually wake you up at night

⬜ A request_id passed through every service

💬 This track goes first because it makes every other track cheaper: any change is visible, any incident takes minutes to investigate instead of "going by gut feeling."

📖 Full article: how developers can monitor production — and why it matters.

Track 2. Infrastructure and deployment

⬜ CI/CD on the most frequently deployed service

⬜ Configuration in git, not on servers

⬜ A reference service template: structure, Dockerfile, healthcheck, pipeline

⬜ Rule: new services only from the template; old ones catch up as you touch them

💬 The goal of this track is for deployment to stop being an event and stop being one person's knowledge. The success marker: a Friday deploy scares no one.

📖 Full article: infrastructure and deployment — order by iteration.

Track 3. Architecture

⬜ Draw the system "as is" — honestly, without prettifying

⬜ Find the boundaries: what is genuinely a separate service and what got glued together by accident

⬜ Start a decision log — short records, half a page: what was decided and why

⬜ Tech debt — an explicit list with a price tag: what it blocks, what it slows down, what a fix costs

💬 The most common mistake is to start by rewriting. Start by writing down: half of the "architecture problems" dissolve once the system is described as it actually is.

📖 Full article: architecture — write it down before rewriting.

Track 4. People

⬜ Bus factor from the Track 0 map: wherever knowledge lives in one head — pair people up or document first

⬜ Onboarding: working environment within a day, first production task within a week

⬜ Hiring: write the job post from the project's real tasks, not from a list of technologies

⬜ Explicit ownership: every service has a name next to it

💬 Order among people matters more than order in code: code with a bus factor of 1 is not an asset, it's a hostage. And the other way around — a team where knowledge flows freely can fix any code.

📖 Full article: people — bus factor, expectations and day one.

Track 5. Processes

⬜ Planning that survives a week: a short cycle, a visible backlog

⬜ A release rhythm — predictable, boring, documented

⬜ Incidents: blameless reviews with one action item each

⬜ Meetings — audit them: each one either produces decisions or dies

💬 Processes come last not because they don't matter, but because without light (Track 1) and hands on the wheel (Track 2) any process is theater.

How to tell the order has arrived

Not by pretty dashboards. By how the team feels:

deploying is not scary,
a new person is useful in their first week,
an incident is a procedure, not a panic,
the question "why is it like this here?" has a written answer.

You'll never reach perfection — and you don't need to. It's enough that every week feels a little calmer than the last one.

This is the map of the series: links in the tracks will come alive as articles are published. Want a specific track covered sooner — tell me, the queue is flexible.

Originally published at mind.mansur.expert.