How to Monitor gRPC Services with Vigilmon

#monitoring #grpc #microservices #api

How to Monitor gRPC Services with Vigilmon

gRPC services communicate over HTTP/2 with Protocol Buffers — they're fast and efficient, but harder to monitor than REST APIs. Traditional uptime checkers can't speak gRPC protocol. Vigilmon monitors your gRPC services through HTTP-based health check endpoints and grpc-gateway proxies.

The gRPC Monitoring Challenge

Standard HTTP monitors can't send gRPC requests. But you have several practical options:

gRPC Health Checking Protocol — expose a standard HTTP endpoint alongside gRPC
grpc-gateway — proxy your gRPC service behind a REST/HTTP gateway
Envoy sidecar — expose admin interface for health metrics
Separate health HTTP endpoint — add a simple HTTP server to your gRPC service

Option 1: gRPC Health Checking Protocol (Recommended)

The gRPC Health Checking Protocol defines a standard grpc.health.v1.Health/Check RPC. Expose this via a gRPC-to-HTTP transcoding proxy or a separate HTTP endpoint.

With grpc-health-probe HTTP wrapper

Many gRPC services run grpc-health-probe as a sidecar and expose its result via HTTP:

// Go example: HTTP wrapper for gRPC health check
http.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
    conn, err := grpc.Dial(":50051", grpc.WithInsecure())
    if err != nil {
        http.Error(w, "gRPC unreachable", 503)
        return
    }
    defer conn.Close()

    client := grpc_health_v1.NewHealthClient(conn)
    resp, err := client.Check(context.Background(), &grpc_health_v1.HealthCheckRequest{})
    if err != nil || resp.Status != grpc_health_v1.HealthCheckResponse_SERVING {
        http.Error(w, "unhealthy", 503)
        return
    }
    w.WriteHeader(200)
    w.Write([]byte(`{"status":"ok"}`))
})

Monitor: https://yourservice.com/healthz

With grpc-gateway

If you use grpc-gateway, your HTTP/JSON endpoints mirror your gRPC methods. Add a health-check endpoint:

// In your .proto file
service HealthService {
  rpc Check(HealthCheckRequest) returns (HealthCheckResponse) {
    option (google.api.http) = {
      get: "/v1/health"
    };
  }
}

This exposes GET /v1/health as a standard HTTP endpoint you can monitor with Vigilmon.

Option 2: Separate Lightweight HTTP Server

The simplest approach: run a tiny HTTP server alongside your gRPC server:

// main.go
go func() {
    mux := http.NewServeMux()
    mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        // Optionally check internal state here
        json.NewEncoder(w).Encode(map[string]string{"status": "ok"})
    })
    http.ListenAndServe(":8080", mux)  // gRPC on :50051, HTTP health on :8080
}()

# Python example (FastAPI alongside gRPC)
app = FastAPI()

@app.get("/health")
def health():
    return {"status": "ok"}

# Run uvicorn on port 8080 alongside grpc server on 50051

In Kubernetes, expose port 8080 separately and monitor the health endpoint via your ingress.

Option 3: Envoy Admin Interface

If you use Envoy as a sidecar or proxy:

GET http://localhost:9901/ready

For an internal monitoring setup, expose this at a stable URL and monitor with Vigilmon:

https://your-envoy-proxy.internal/ready

Setting Up Vigilmon for gRPC Services

Vigilmon setup:

URL: https://yourservice.com/health (or /healthz, /ready)
Expected status: 200
Keyword: "ok" or "status":"SERVING"
Timeout: 10 seconds (gRPC services can take a moment on cold start)
Interval: 1–2 minutes for critical services

Deep Health Check: Test a Real RPC Call

For the most comprehensive monitoring, implement a dedicated health check RPC that exercises your service:

func (s *HealthServer) Check(ctx context.Context, req *healthpb.HealthCheckRequest) (*healthpb.HealthCheckResponse, error) {
    // Test your DB connection
    if err := s.db.PingContext(ctx); err != nil {
        return &healthpb.HealthCheckResponse{
            Status: healthpb.HealthCheckResponse_NOT_SERVING,
        }, nil
    }

    return &healthpb.HealthCheckResponse{
        Status: healthpb.HealthCheckResponse_SERVING,
    }, nil
}

Wrap this in your HTTP health endpoint to expose to Vigilmon.

Kubernetes Integration

In Kubernetes, configure livenessProbe and readinessProbe for your gRPC pods:

livenessProbe:
  httpGet:
    path: /health
    port: 8080
  initialDelaySeconds: 10
  periodSeconds: 30

readinessProbe:
  exec:
    command: ["/bin/grpc-health-probe", "-addr=:50051"]
  initialDelaySeconds: 5

The Kubernetes probe handles internal health; Vigilmon monitors your externally-facing gRPC endpoints.

What to Alert On

503 Service Unavailable — gRPC service is NOT_SERVING
504 Gateway Timeout — your HTTP health wrapper can't reach gRPC
500 Internal Server Error — health check threw an exception
Missing keyword — response is 200 but status isn't "ok" (partial degradation)

Summary

Approach	Complexity	Recommendation
HTTP wrapper for gRPC health	Low	Best for most services
grpc-gateway endpoint	Medium	If you already use grpc-gateway
Separate HTTP health server	Low	Easiest to add to existing services
Envoy admin interface	Medium	If using Envoy as sidecar