Service Discovery at 50K RPS: Hardening Consul for Production Go Microservices

#go #backend #concurrency #performance

The Problem: DNS Round-Robin Isn't Enough

When our payment gateway started scaling past 30K requests per second, DNS-based service discovery became the bottleneck. Stale A-records caused 3-5 second failover delays during deployments, and we had no circuit breaker visibility into unhealthy upstream instances.

HashiCorp Consul solved this with real-time service health propagation and native Go client libraries. But integrating it into a high-concurrency runtime revealed sharp edges around connection pooling, blocking queries, and goroutine safety.

Consul Client: The Goroutine-Safe Baseline

Consul's official Go client (github.com/hashicorp/consul/api) is goroutine-safe by design, but you must configure connection limits to prevent file descriptor exhaustion:

package main

import (
    "fmt"
    "time"
    consulapi "github.com/hashicorp/consul/api"
)

func NewConsulClient() (*consulapi.Client, error) {
    config := consulapi.DefaultConfig()
    config.Address = "localhost:8500"
    config.HttpClient.Timeout = 3 * time.Second

    // CRITICAL: Bound connection pool to prevent FD leaks
    config.Transport.MaxIdleConns = 100
    config.Transport.MaxIdleConnsPerHost = 20

    return consulapi.NewClient(config)
}

Without MaxIdleConnsPerHost, we saw 12K open connections under load—each blocking query holds a persistent HTTP connection.

Health Check Tuning: The 10-Second Penalty

Consul's default health check interval (10s) meant a crashing service stayed in the load balancer pool for up to 10 seconds, causing 500 errors:

// agent/service_register.json
{
  "Name": "payment-gateway",
  "Port": 8080,
  "Check": {
    "HTTP": "http://localhost:8080/health",
    "Interval": "2s",          // Reduced from 10s default
    "Timeout": "1s",
    "DeregisterCriticalServiceAfter": "30s"
  }
}

Reducing interval to 2s cut failover time by 80%, but CPU usage on Consul agents increased by 15%. The trade-off depends on your SLA budget.

Production Pitfall: Blocking Query Goroutine Leaks

Consul's blocking queries (long-polling for config changes) require careful lifecycle management:

func WatchServiceHealth(client *consulapi.Client, service string, stopCh <-chan struct{}) {
    queryOpts := &consulapi.QueryOptions{
        WaitTime: 5 * time.Minute, // Long-poll timeout
    }

    for {
        select {
        case <-stopCh:
            return // Prevent goroutine leak on shutdown
        default:
            services, meta, err := client.Health().Service(
                service, "", true, queryOpts,
            )
            if err != nil {
                time.Sleep(1 * time.Second)
                continue
            }

            queryOpts.WaitIndex = meta.LastIndex
            processHealthUpdates(services)
        }
    }
}

Without the stopCh select, we leaked 200+ goroutines during blue-green deployments—each holding a blocked HTTP connection.

Benchmark: Native Consul vs. DNS Lookup

Measuring service resolution latency with Go's testing.B:

func BenchmarkConsulServiceDiscovery(b *testing.B) {
    client, _ := NewConsulClient()

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        services, _, _ := client.Health().Service(
            "payment-gateway", "", true, nil,
        )
        _ = services[0].Service.Address
    }
}

// Results on 3-node Consul cluster:
// BenchmarkConsulServiceDiscovery-8   5000   285432 ns/op   1024 B/op   12 allocs/op
// vs. net.LookupHost:                 8000   175234 ns/op    512 B/op    8 allocs/op

Consul adds ~110µs per lookup but eliminates DNS caching issues. For request paths already doing database queries (2-5ms), this overhead is negligible.

The Unresolved Trade-Off: Consistency vs. Availability

Consul's Raft consensus guarantees strong consistency, but during network partitions, minority nodes reject writes. For read-heavy service discovery, enabling stale reads trades consistency for availability:

queryOpts := &consulapi.QueryOptions{
    AllowStale: true, // Read from followers, tolerate slight staleness
}

This prevented cascading failures when our primary datacenter's Consul leader went down, but we observed 200-500ms staleness in service registration propagation.

Is eventual consistency acceptable for your service topology, or does every request need the absolute latest health state? That's the operational question Consul forces you to answer explicitly.

DEV Community