DEV Community: Pires Aurélio

I Added Prometheus to My Go Job Scraper and It Changed How I Think About Production

Pires Aurélio — Sat, 03 Oct 2026 23:27:21 +0000

I Added Prometheus to My Go Job Scraper and It Changed How I Think About Production

I was building a Go scraper for Google Jobs (https://github.com/Aureliopires186/go-jobs-tracker) and I realized: if it runs in production and nobody can observe it, it doesn't exist.

So I added Prometheus. Here's the exact minimal setup that made my project look senior-level.

Why Metrics Matter

Without metrics, you are blind. You don't know:

How many jobs you scraped?
Is it failing silently?
How long does it take?

Prometheus solves that.

The Full Implementation

Step A: Create metrics/metrics.go
This is my live file: https://github.com/Aureliopires186/go-jobs-tracker/tree/main/metrics


go
package metrics

import (
    "github.com/prometheus/client_golang/prometheus"
    "github.com/prometheus/client_golang/prometheus/promauto"
)

var (
    JobsScraped = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "jobs_scraped_total",
            Help: "Total number of jobs scraped per engine and status",
        },
        []string{"engine", "status"},
    )

    ScrapeDuration = promauto.NewHistogramVec(
        prometheus.HistogramOpts{
            Name:    "scrape_duration_seconds",
            Help:    "Duration of scrape jobs in seconds",
            Buckets: prometheus.DefBuckets,
        },
        []string{"engine"},
    )

    ErrorsTotal = promauto.NewCounterVec(
        prometheus.CounterOpts{
            Name: "scraper_errors_total",
            Help: "Total scraper errors by type",
        },
        []string{"type"},
    )
)
*Step B: Expose /metrics and /healthz in `main.go`*
package main

import (
    "net/http"
    "time"
    "github.com/prometheus/client_golang/prometheus/promhttp"
    "github.com/Aureliopires186/go-jobs-tracker/metrics"
)

func main() {
    // Start metrics server
    go func() {
        http.Handle("/metrics", promhttp.Handler())
        http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
            w.WriteHeader(200)
            w.Write([]byte("ok"))
        })
        http.ListenAndServe(":2112", nil)
    }()

    // Your scraping logic
    start := time.Now()
    // ... scrape ...
    duration := time.Since(start)

    metrics.JobsScraped.WithLabelValues("google_jobs", "success").Inc()
    metrics.ScrapeDuration.WithLabelValues("google_jobs").Observe(duration.Seconds())
}
*Step C: Install*
go get github.com/prometheus/client_golang@v1.19.0
go mod tidy
go run main.go
curl http://localhost:2112/metrics
You should see jobs_scraped_total
3. Docker + Prometheus Ready

My Dockerfile already exposes 2112.

`prometheus.yml`:
scrape_configs:
    - job_name: 'go-jobs-tracker'
    static_configs:
            - targets: ['localhost:2112']
Now you can create Grafana dashboards with `jobs_scraped_total` and `scrape_duration_seconds`.

4. What I Learned

Senior code is not about more features. It's about observability, health checks, and knowing when your system fails before the client does.

Full code: https://github.com/Aureliopires186/go-jobs-tracker

Follow me:
- LinkedIn: https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
- Blog: https://blogspotangola.blogspot.com/
- CoderLegion: https://coderlegion.com/user/Pires+Aurlio

If you are building scrapers in Go, add Prometheus on day 1. Your future self will thank you.

github.com/Aureliopires186/go-jobs-tracker/metrics

I Built a Production-Ready Job Queue in Go from Scratch - Goroutines, Postgres and Clean Architecture published: true

Pires Aurélio — Thu, 01 Oct 2026 11:50:45 +0000

I'm Pires Aurélio , a Go Backend Engineer from Luanda, Angola. I build production systems in public.

For 2 years I built scrapers and REST APIs. I got tired of tutorials that only show "Hello World" with Gin. No jobs, no retries, no real problems.

So I built a real one: A Production-Ready Background Job Queue in Go.

My Public Work:

Main Scraper Framework (Python AsyncIO): https://github.com/Aureliopires186/py-websites-scraper
Go Production Examples: https://github.com/Aureliopires186/go-serpapi-examples
Blog - REST API with Go (Gin): https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html
LinkedIn / Portfolio: https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385

What We Are Building

A REST API where you POST a task, it goes to Postgres, and Go workers process it in background with retries, rate limiting and graceful shutdown. Like Sidekiq / Celery, but in pure Go.

Stack: Go + Gin + Postgres + Goroutines + Docker
Features:

REST API with Clean Architecture
Postgres as queue (no Redis needed)
Worker Pool with 5 Goroutines
Graceful Shutdown
Retry with exponential backoff

Architecture (The Key for Recruiters)

Client -> POST /jobs -> API (Gin) -> Postgres (jobs table) -> Worker Pool -> Process -> Update Status

Why Postgres and not Redis? For portfolio and for cost. One less service. And it proves you understand ACID, transactions and FOR UPDATE SKIP LOCKED.

Database Schema - The Foundation


sql
CREATE TABLE jobs (
  id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
  payload JSONB NOT NULL,
  status VARCHAR(20) DEFAULT 'pending',
  attempts INT DEFAULT 0,
  max_attempts INT DEFAULT 3,
  created_at TIMESTAMPTZ DEFAULT NOW(),
  next_retry_at TIMESTAMPTZ DEFAULT NOW()
);

CREATE INDEX idx_jobs_status_next_retry ON jobs(status, next_retry_at);
This index `idx_jobs_status_next_retry` is critical. It makes the worker query O(log n) even with 1 million jobs.

3. The Worker - The Heart of the System

This is what 90% of tutorials don't teach. This is what makes you Senior.
package worker

import (
  "context"
  "log"
  "math"
  "time"
)

func (w *Worker) Start(ctx context.Context) {
  for i := 0; i < w.concurrency; i++ {
    go func(workerID int) {
      log.Printf("Worker %d started", workerID)
      for {
        select {
        case <-ctx.Done():
          log.Printf("Worker %d shutting down gracefully", workerID)
          return
        default:
          job, err := w.repo.FetchPendingJob(ctx) // Uses FOR UPDATE SKIP LOCKED
          if err!= nil {
            time.Sleep(1 * time.Second)
            continue
          }
          if job == nil {
            time.Sleep(500 * time.Millisecond)
            continue
          }
          w.processWithRetry(ctx, job)
        }
      }
    }(i)
  }
}

func (w *Worker) processWithRetry(ctx context.Context, job *Job) {
  err := w.processor.Process(job)
  if err!= nil {
    job.Attempts++
    if job.Attempts >= job.MaxAttempts {
      w.repo.MarkFailed(ctx, job.ID)
      log.Printf("Job %s failed permanently after %d attempts", job.ID, job.Attempts)
    } else {
      backoff := time.Duration(math.Pow(2, float64(job.Attempts))) * time.Second
      w.repo.MarkForRetry(ctx, job.ID, time.Now().Add(backoff))
      log.Printf("Job %s failed, retrying in %v", job.ID, backoff)
    }
    return
  }
  w.repo.MarkDone(ctx, job.ID)
}
*What recruiters see here:*
1. `FOR UPDATE SKIP LOCKED` - You know how to prevent double-processing
2. `context.Context` - You know Graceful Shutdown
3. Exponential Backoff - You think about production resilience

4. The Repository Query (The Secret Sauce)
func (r *JobRepo) FetchPendingJob(ctx context.Context) (*Job, error) {
  query := `
    SELECT id, payload, attempts, max_attempts
    FROM jobs
    WHERE status = 'pending' AND next_retry_at <= NOW()
    ORDER BY created_at ASC
    LIMIT 1
    FOR UPDATE SKIP LOCKED`

  // This SKIP LOCKED is GOLD. Two workers will never get the same job.
  //...
}
5. REST API with Gin - Clean Handler
func (h *JobHandler) CreateJob(c *gin.Context) {
  var req CreateJobRequest
  if err := c.ShouldBindJSON(&req); err!= nil {
    c.JSON(400, gin.H{"error": err.Error()})
    return
  }
  job, err := h.service.Enqueue(c.Request.Context(), req.Payload)
  if err!= nil {
    c.JSON(500, gin.H{"error": "failed to enqueue job"})
    return
  }
  c.JSON(201, job)
}

func (h *JobHandler) ListJobs(c *gin.Context) {
  jobs, _ := h.service.List(c.Request.Context())
  c.JSON(200, jobs)
}
No business logic in handler. That's Clean Architecture. Handler -> Service -> Repo.

6. Docker Compose - One Command Run
version: '3.8'
services:
  api:
    build:.
    ports: ["8080:8080"]
    environment:
      - DATABASE_URL=postgres://postgres:postgres@db:5432/jobs?sslmode=disable
    depends_on: [db]
  db:
    image: postgres:15
    environment:
      - POSTGRES_DB=jobs
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
    ports: ["5432:5432"]
`docker-compose up` and the whole system is up.

7. What I Learned Building This in Luanda

1. *Concurrency is not parallelism:* Goroutines are cheap (2KB), but Postgres connections are not. I use a pgx pool with max 10 connections.
2. *Observability from day 1:* I added JobID in every log. In a real company, this would be Prometheus + Grafana.
3. *Failure is the default:* 80% of the code is handling retries, shutdown, and errors. 20% is the happy path.

Conclusion

Any junior can build a CRUD. A mid-level builds a system that doesn't break when 10,000 jobs arrive at once.

This project proves:
- You understand Go concurrency
- You understand databases beyond SELECT *
- You think about failure, not just success

Full code of this queue is coming to my GitHub next week. Follow for Part 2: Adding Rate Limiter, Metrics and Dead Letter Queue.

Built in public from Luanda, Angola 🇦🇴

If you are a recruiter looking for a Go backend engineer who ships production-ready systems, let's talk.

---
Examples using public technologies. All code is my own. Not affiliated with any provider.

Author : pires Aurélio 
E-mail: aureliopires186@gmail.com 
https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
https://github.com/Aureliopires186
https://coderlegion.com/user/Pires+Aurlio
https://blogspotangola.blogspot.com/2026/09/building-production-ready-google-jobs.html

I Migrated from Python Scrapers to Go and Search APIs - Complete Guide published: true

Pires Aurélio — Tue, 29 Sep 2026 18:17:18 +0000

•Observation :
Examples using public third-party APIs. Not affiliated with or endorsed by any API provider. All trademarks belong to their respective owners.

I'm Pires Aurélio , a Go/Python developer from Angola. I build scrapers in production and document everything in public.

My work:

Main scraper (Python/AsyncIO): https://github.com/Aureliopires186/py-websites-scraper
Go SerpApi Examples: https://github.com/Aureliopires186/go-serpapi-examples
Jobs Scraper Tracker: https://github.com/Aureliopires186/-serpApi-go-jobs-scraper/issues/3
Blog - How to build REST API in Go: https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html
LinkedIn: https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385

For 2 years I scraped sites with pure Python/Go. Then I hit Google. Everything broke.

This tutorial is how I solved it with Go + SerpApi.

1. Why My Python Scraper Failed on Google

My repo py-websites-scraper uses asyncio + aiohttp + readability-lxml. Great for blogs.

But for Google Jobs / Shopping:

CAPTCHA in 10 requests
429 blocks
Selectors change weekly

I was spending 80% time fighting anti-bot, 20% building features. Wrong.

2. Why SerpApi + Go Is The Stack

SerpApi handles proxies, CAPTCHAs, parsing. You get clean JSON.
Go handles concurrency perfectly.

My stack now: Go + SerpApi + Gin REST API (as I teach in my blog)

3. Setup (2 minutes)


bash
go mod init go-serpapi-demo
go get github.com/serpapi/google-search-results-golang
go get github.com/gin-gonic/gin
export SERPAPI_KEY=your_free_key_from_serpapi.com

#. Example 1: Google Jobs - The Code I Use

package main

import (
  "fmt"
  "log"
  "os"

  g "github.com/serpapi/google-search-results-golang"
)

// Só pegamos o que vamos usar - tipo seguro
type Job struct {
  Title string `json:"title"`
  CompanyName string `json:"company_name"`
  Via string `json:"via"`
}

func main() {
  apiKey := os.Getenv("SERPAPI_KEY")
  if apiKey == "" {
    log.Fatal("SERPAPI_KEY not set")
  }

  params := map[string]string{
    "engine": "google_jobs",
    "q": "Go Developer",
    "location": "Remote",
    "hl": "en",
  }

  search := g.NewGoogleSearch(params, apiKey)
  results, err := search.GetJSON()
  if err!= nil {
    log.Fatalf("SerpApi request failed: %v", err)
  }

  // FORMA SEGURA que o Mike pediu
  jobsRaw, ok := results["jobs_results"].([]interface{})
  if!ok {
    log.Printf("No jobs_results in response: %+v", results)
    return
  }

  if len(jobsRaw) == 0 {
    fmt.Println("No jobs found")
    return
  }

  for _, j := range jobsRaw {
    jobMap, ok := j.(map[string]interface{})
    if!ok {
      continue // pula item estranho sem quebrar
    }
    // print seguro
    fmt.Printf("[%v] %v - %v\n", jobMap["via"], jobMap["title"], jobMap["company_name"])
  }
}

#. Example 2: Google Shopping Price Tracker

package main

import (
  "fmt"
  "log"
  "os"

  g "github.com/serpapi/google-search-results-golang"
)

func main() {
  apiKey := os.Getenv("SERPAPI_KEY")
  if apiKey == "" {
    log.Fatal("SERPAPI_KEY not set")
  }

  params := map[string]string{
    "engine": "google_shopping",
    "q": "golang book",
  }

  search := g.NewGoogleSearch(params, apiKey)

  // NUNCA ignore erro - correção do Mike
  results, err := search.GetJSON()
  if err!= nil {
    log.Fatalf("failed to get shopping results: %v", err)
  }

  shoppingResults, ok := results["shopping_results"].([]interface{})
  if!ok {
    log.Printf("No shopping_results found: %+v", results)
    return
  }

  fmt.Printf("Found %d products\n", len(shoppingResults))
  for _, item := range shoppingResults {
    if m, ok := item.(map[string]interface{}); ok {
      fmt.Printf("- %v: %v\n", m["title"], m["price"])
    }
  }
}


By:Pires Aurélio 
Contact: aureliopires186@gmail.com 
https://blogspotangola.blogspot.com/?m=1
https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385

I Open-Sourced 5 Real-World API Integration Examples with Go (Gin) — Here's What I Learned

Pires Aurélio — Wed, 23 Sep 2026 19:18:11 +0000

I Open-Sourced 5 Real-World API Integration Examples with Go (Gin) — Here's What I Learned

•Observation :
Examples using public third-party APIs. Not affiliated with or endorsed by any API provider. All trademarks belong to their respective owners.

I love Go for its simplicity. But finding clean, real-world examples of how to use Search APIs with Go is hard. So I decided to build it myself.

I'm excited to share my new open-source project:

👉 https://github.com/Aureliopires186/go-serpapi-examples

The goal is simple: help Go developers in Africa and around the world build products with real-time data without dealing with scraping, proxies, or captchas.

What's inside?

The repo is not just one example, it's a collection of ready-to-use REST APIs:

✅ Google Jobs API – Build your own job board (perfect for remote jobs)
✅ Google Maps API – Location search & place details
✅ Google Search API – SERP tracking tool
✅ All built with clean Go, Gin, and best practices

Each example has a README.md with step-by-step instructions.

Why SerpApi + Go?

I tested scraping myself. It broke every week. SerpApi just works and returns clean JSON. For Go developers, that's gold. You focus on go run, not on fixing parsers.

Full Tutorial

I wrote a detailed breakdown of how I built the Jobs API on my blog:

📖 Read the full tutorial: https://blogspotangola.blogspot.com/2026/09/how-to-create-simple-rest-api-with.html

What's next?

This is just the start. I want to add more APIs (Flights, Shopping, Scholar) and create video tutorials.

If this project helped you, please:

⭐ Star the repo to support the work
💬 Open an issue telling me what example you want next
🔗 Connect with me, I'm documenting my journey to becoming a Developer Advocate

Let's connect:
LinkedIn: https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
My Blog: https://blogspotangola.blogspot.com/

Thanks for reading! What API would you build first?

Author : pires Aurélio
E-mail: aureliopires186@gmail.com
https://coderlegion.com/user/Pires+Aurlio

https://github.com/Aureliopires186
https://www.linkedin.com/in/pires-aur%C3%A9lio-511330385
https://blogspotangola.blogspot.com/2026/09/building-production-ready-google-jobs.html

Search Google Jobs in Real-Time with Go 🚀

Pires Aurélio — Wed, 23 Sep 2026 10:13:52 +0000

•Observation :
Examples using public third-party APIs. Not affiliated with or endorsed by any API provider. All trademarks belong to their respective owners.

Hey devs! 👋

Tired of manual scraping to get Google Jobs listings? 🤔

I built a Go example that uses the SerpApi to fetch Google Jobs data in real-time! 👀

🔍 What this tutorial covers:

How to set up SerpApi
Searching jobs with Go
Parsing results to JSON
Deploying the API to the cloud

👉 Links:

💡 Need help with automation? Feel free to reach out!