I Added Prometheus to My Go Job Scraper and It Changed How I Think About Production

I Added Prometheus to My Go Job Scraper and It Changed How I Think About Production

I was building a Go scraper for Google Jobs (https://github.com/Aureliopires186/go-jobs-tracker) and I realized: if it runs in production and nobody can observe it, it doesn't exist.

So I added Prometheus. Here's the exact minimal setup that made my project look senior-level.

1. Why Metrics Matter

Without metrics, you are blind. You don't know:

• How many jobs you scraped?
• Is it failing silently?
• How long does it take?

Prometheus solves that.

1. The Full Implementation

Step A: Create metrics/metrics.go
This is my live file: https://github.com/Aureliopires186/go-jobs-tracker/tree/main/metrics

go
package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    JobsScraped = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "jobs_scraped_total",
            Help: "Total number of jobs scraped per engine and status",
        },
        []string{"engine", "status"},
    )

    ScrapeDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "scrape_duration_seconds",
            Help:    "Duration of scrape jobs in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"engine"},
    )

    ErrorsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "scraper_errors_total",
            Help: "Total scraper errors by type",
        },
        []string{"type"},
    )
)
*Step B: Expose /metrics and /healthz in `main.go`*
package main

import (
    "net/http"
    "time"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/Aureliopires186/go-jobs-tracker/metrics"
)

func main() {
    // Start metrics server
    go func() {
        http.Handle("/metrics", promhttp.Handler())
        http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
            w.WriteHeader(200)
            w.Write([]byte("ok"))
        })
        http.ListenAndServe(":2112", nil)
    }()

    // Your scraping logic
    start := time.Now()
    // ... scrape ...
    duration := time.Since(start)

    metrics.JobsScraped.WithLabelValues("google_jobs", "success").Inc()
    metrics.ScrapeDuration.WithLabelValues("google_jobs").Observe(duration.Seconds())
}
*Step C: Install*
go get github.com/prometheus/client_golang@v1.19.0
go mod tidy
go run main.go
curl http://localhost:2112/metrics
You should see jobs_scraped_total
3. Docker + Prometheus Ready

My Dockerfile already exposes 2112.

`prometheus.yml`:
scrape_configs:
    - job_name: 'go-jobs-tracker'
    static_configs:
            - targets: ['localhost:2112']
Now you can create Grafana dashboards with `jobs_scraped_total` and `scrape_duration_seconds`.

4. What I Learned

Senior code is not about more features. It's about observability, health checks, and knowing when your system fails before the client does.

Full code: https://github.com/Aureliopires186/go-jobs-tracker

Follow me:
- LinkedIn: https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
- Blog: https://blogspotangola.blogspot.com/
- CoderLegion: https://coderlegion.com/user/Pires+Aurlio

If you are building scrapers in Go, add Prometheus on day 1. Your future self will thank you.

github.com/Aureliopires186/go-jobs-tracker/metrics

Comments

[Doyoon Kim]

Seeing you instrument the scraper with Prometheus really highlights how even a simple batch job benefits from real‑time metrics—especially the latency histograms you added for request durations. In my recent work on an AI‑powered data pipeline, exposing custom process‑level gauges for queue depth let us spot back‑pressure before it affected downstream services. Have you experimented with alerting on scrape error rates, or do you rely on downstream monitori…