DEV Community: Ayi NEDJIMI

Building a Zero-Trust API Gateway with Certificate-Based Auth

Ayi NEDJIMI — Sat, 31 Oct 2026 10:02:46 +0000

Most API gateways authenticate callers with API keys or JWTs — shared secrets that get stolen, rotated lazily, or leaked into logs. Certificate-based mutual TLS (mTLS) eliminates that class of risk entirely: every client proves cryptographic identity on every request, and there is no bearer token to exfiltrate. Here is how to build a minimal zero-trust gateway in Go that enforces it.

Why API Keys Keep Failing

API keys work until they don't. They're long-lived, embedded in environment variables, accidentally committed to repos, passed through logging middleware, and shared across services. The OWASP API Security Top 10 consistently ranks broken authentication as the top risk, and API key misuse is a major contributor.

A client certificate solves these problems at the transport layer:

Certificates can't be replicated without the private key
Revocation is built in (via CRL or OCSP)
Identity is tied to a cryptographic keypair, not a string that can be copied with Ctrl+C

The tradeoff: you need a PKI. For an internal API gateway, a self-signed CA is entirely appropriate.

Setting Up a Minimal CA

Before writing Go code, we need certificates. OpenSSL gives us a CA, a server cert, and a client cert:

# Generate the CA key and self-signed certificate
openssl genrsa -out ca.key 4096
openssl req -x509 -new -nodes -key ca.key -sha256 -days 3650 \
  -subj "/CN=Internal CA" -out ca.crt

# Server certificate
openssl genrsa -out server.key 2048
openssl req -new -key server.key -subj "/CN=api.internal" -out server.csr
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out server.crt -days 365 -sha256

# Client certificate (one per service or team)
openssl genrsa -out client-service-a.key 2048
openssl req -new -key client-service-a.key \
  -subj "/CN=service-a/O=backend-team" -out client-service-a.csr
openssl x509 -req -in client-service-a.csr -CA ca.crt -CAkey ca.key \
  -CAcreateserial -out client-service-a.crt -days 365 -sha256

The /CN and /O fields in the client cert are your identity claims — we'll extract them in the gateway to make routing decisions.

Building the Gateway in Go

The core of a zero-trust gateway is tls.Config with ClientAuth: tls.RequireAndVerifyClientCert. Go's standard library handles the certificate verification chain; we extract the identity and pass it downstream.

package main

import (
    "crypto/tls"
    "crypto/x509"
    "fmt"
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
    "os"
    "strings"
)

func loadCA(path string) *x509.CertPool {
    pool := x509.NewCertPool()
    pem, err := os.ReadFile(path)
    if err != nil {
        log.Fatalf("reading CA: %v", err)
    }
    if !pool.AppendCertsFromPEM(pem) {
        log.Fatal("failed to parse CA cert")
    }
    return pool
}

func identityMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        if r.TLS == nil || len(r.TLS.PeerCertificates) == 0 {
            http.Error(w, "client certificate required", http.StatusUnauthorized)
            return
        }
        cert := r.TLS.PeerCertificates[0]
        cn := cert.Subject.CommonName
        org := ""
        if len(cert.Subject.Organization) > 0 {
            org = cert.Subject.Organization[0]
        }
        r.Header.Set("X-Client-CN", cn)
        r.Header.Set("X-Client-Org", org)
        next.ServeHTTP(w, r)
    })
}

var routePolicy = map[string]string{
    "/admin/":    "ops-team",
    "/internal/": "backend-team",
}

func authzMiddleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        org := r.Header.Get("X-Client-Org")
        for prefix, required := range routePolicy {
            if strings.HasPrefix(r.URL.Path, prefix) && org != required {
                http.Error(w, "forbidden", http.StatusForbidden)
                return
            }
        }
        next.ServeHTTP(w, r)
    })
}

func main() {
    upstream, _ := url.Parse("http://127.0.0.1:8080")
    proxy := httputil.NewSingleHostReverseProxy(upstream)

    mux := http.NewServeMux()
    mux.Handle("/", identityMiddleware(authzMiddleware(proxy)))

    cert, err := tls.LoadX509KeyPair("server.crt", "server.key")
    if err != nil {
        log.Fatalf("loading server cert: %v", err)
    }

    srv := &http.Server{
        Addr:    ":8443",
        Handler: mux,
        TLSConfig: &tls.Config{
            Certificates: []tls.Certificate{cert},
            ClientCAs:    loadCA("ca.crt"),
            ClientAuth:   tls.RequireAndVerifyClientCert,
            MinVersion:   tls.VersionTLS13,
        },
    }

    fmt.Println("Gateway listening on :8443 (mTLS enforced)")
    log.Fatal(srv.ListenAndServeTLS("", ""))
}

This gateway:

Terminates TLS and verifies the client certificate against your CA
Rejects any request without a valid cert with 401 Unauthorized
Injects X-Client-CN and X-Client-Org headers before proxying upstream
Enforces TLS 1.3 minimum — no downgrade possible
Checks route-level authorization based on the client organization field

Testing the Gateway

With the certs generated, you can test immediately with curl:

# Should succeed — valid client cert
curl --cert client-service-a.crt --key client-service-a.key \
     --cacert ca.crt https://localhost:8443/internal/status

# Should fail — no cert
curl --cacert ca.crt https://localhost:8443/internal/status
# curl: (56) OpenSSL SSL_read: error:... alert handshake failure

# Should fail — wrong org for /admin/
curl --cert client-service-a.crt --key client-service-a.key \
     --cacert ca.crt https://localhost:8443/admin/users
# HTTP/2 403 forbidden

The TLS handshake failure for the no-cert case happens before any HTTP code runs — the connection is dropped at the transport layer, not the application layer. That is exactly what zero-trust means in practice.

Revocation and Certificate Rotation

Revocation is where cert-based auth gets operationally expensive if you're not deliberate about it.

Short-lived certs are the simplest approach: issue certs valid for 24-72 hours and automate renewal. A compromised cert expires before most incident response workflows could react. Tools like SPIFFE/SPIRE handle this in Kubernetes environments; for simpler setups, a cron job plus a renewal endpoint works fine.

CRL approach: if you need immediate revocation, host a certificate revocation list and embed the CRL distribution point URL at issuance time. Go's x509 package checks CRLs if you configure a custom VerifyPeerCertificate callback.

Serial blocklist: the simplest fast-revocation mechanism is a Redis set of revoked serial numbers checked in identityMiddleware. Not cryptographically rigorous, but operational within minutes of detecting a compromise.

We maintain free security hardening checklists covering PKI hygiene in depth — OCSP stapling, CRL caching, rotation timelines, and cert pinning tradeoffs.

The Takeaway

mTLS-based API gateways eliminate the entire API key leakage risk category. The Go implementation is compact — standard library tls.Config does the verification work. The hard part is operational: issuing certs, rotating them before expiry, and handling revocation when something goes wrong.

Start with long-lived certs and a manual blocklist. Automate rotation only after you've validated the issuance pipeline end-to-end. The zero-trust claim here is precise: every request carries cryptographic proof of identity, verified on every connection, with no session state to steal.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

How I Cut API Response Times by 60% with Go Caching Strategies

Ayi NEDJIMI — Fri, 30 Oct 2026 10:01:26 +0000

How I Cut API Response Times by 60% with Go Caching Strategies

A slow API bleeds users. When p99 latency on one of our endpoints crept above 800ms, we had to fix it without refactoring the entire stack. Caching was the answer — but the implementation details determine whether you get 60% gains or a subtle data consistency bug.

The Problem: Every Request Hit the Database

Our setup was straightforward: a Go API, PostgreSQL, and a set of endpoints that fetched reference data on every request. The data changed maybe once a day. We were fetching it thousands of times a minute.

Profiling with pprof made it obvious — 70% of request time was in database roundtrips for data that was effectively static.

// Before: fetching config on every request
func (h *Handler) GetUserDashboard(w http.ResponseWriter, r *http.Request) {
    userID := chi.URLParam(r, "userID")

    // This hit the DB every time — 40-120ms per call
    config, err := h.db.GetSystemConfig(r.Context())
    if err != nil {
        http.Error(w, "internal error", 500)
        return
    }

    user, err := h.db.GetUser(r.Context(), userID)
    if err != nil {
        http.Error(w, "not found", 404)
        return
    }

    render.JSON(w, r, buildDashboard(user, config))
}

Layer 1: In-Memory Cache with TTL

The fastest cache is in-process memory. No network hop, no serialization. For small, frequently-read datasets, a simple TTL cache outperforms Redis on pure latency.

Here's a thread-safe generic in-memory cache I use in production:

package cache

import (
    "sync"
    "time"
)

type entry[V any] struct {
    value     V
    expiresAt time.Time
}

type TTLCache[K comparable, V any] struct {
    mu   sync.RWMutex
    data map[K]entry[V]
    ttl  time.Duration
}

func New[K comparable, V any](ttl time.Duration) *TTLCache[K, V] {
    c := &TTLCache[K, V]{
        data: make(map[K]entry[V]),
        ttl:  ttl,
    }
    go c.janitor()
    return c
}

func (c *TTLCache[K, V]) Get(key K) (V, bool) {
    c.mu.RLock()
    e, ok := c.data[key]
    c.mu.RUnlock()
    if !ok || time.Now().After(e.expiresAt) {
        var zero V
        return zero, false
    }
    return e.value, true
}

func (c *TTLCache[K, V]) Set(key K, value V) {
    c.mu.Lock()
    c.data[key] = entry[V]{value: value, expiresAt: time.Now().Add(c.ttl)}
    c.mu.Unlock()
}

func (c *TTLCache[K, V]) Delete(key K) {
    c.mu.Lock()
    delete(c.data, key)
    c.mu.Unlock()
}

func (c *TTLCache[K, V]) janitor() {
    ticker := time.NewTicker(c.ttl / 2)
    for range ticker.C {
        now := time.Now()
        c.mu.Lock()
        for k, e := range c.data {
            if now.After(e.expiresAt) {
                delete(c.data, k)
            }
        }
        c.mu.Unlock()
    }
}

The RLock/RUnlock pattern is critical: multiple goroutines can read simultaneously. We only take an exclusive write lock on Set and Delete. The janitor goroutine runs every ttl/2 to evict stale entries and prevent unbounded memory growth.

Wiring it into the handler:

type Handler struct {
    db          *Store
    configCache *cache.TTLCache[string, SystemConfig]
}

func NewHandler(db *Store) *Handler {
    return &Handler{
        db:          db,
        configCache: cache.New[string, SystemConfig](5 * time.Minute),
    }
}

func (h *Handler) getSystemConfig(ctx context.Context) (SystemConfig, error) {
    const key = "system_config"
    if cfg, ok := h.configCache.Get(key); ok {
        return cfg, nil
    }
    cfg, err := h.db.GetSystemConfig(ctx)
    if err != nil {
        return SystemConfig{}, err
    }
    h.configCache.Set(key, cfg)
    return cfg, nil
}

Config-fetch latency dropped from ~60ms to ~0.1ms for cache hits.

Layer 2: Singleflight to Prevent Cache Stampedes

There's a failure mode that bites people in production. When a cache entry expires and 200 concurrent requests arrive simultaneously, all 200 miss the cache and hammer the database at once. This is a cache stampede.

The fix is golang.org/x/sync/singleflight:

import "golang.org/x/sync/singleflight"

type Handler struct {
    db          *Store
    configCache *cache.TTLCache[string, SystemConfig]
    sfGroup     singleflight.Group
}

func (h *Handler) getSystemConfig(ctx context.Context) (SystemConfig, error) {
    const key = "system_config"
    if cfg, ok := h.configCache.Get(key); ok {
        return cfg, nil
    }

    result, err, _ := h.sfGroup.Do(key, func() (any, error) {
        cfg, err := h.db.GetSystemConfig(ctx)
        if err != nil {
            return nil, err
        }
        h.configCache.Set(key, cfg)
        return cfg, nil
    })
    if err != nil {
        return SystemConfig{}, err
    }
    return result.(SystemConfig), nil
}

singleflight.Group.Do ensures that only one in-flight call executes the function for a given key. All other callers block and receive the same result when it completes. The stampede is gone, and the database sees one query instead of 200.

Layer 3: HTTP ETags for Unchanged Resources

For endpoints where clients repeatedly fetch the same resource, ETags let you skip serialization and payload transmission entirely when nothing changed:

func (h *Handler) GetCatalog(w http.ResponseWriter, r *http.Request) {
    catalog, version, err := h.getCatalog(r.Context())
    if err != nil {
        http.Error(w, "error", 500)
        return
    }

    etag := fmt.Sprintf(`"%s"`, version)
    w.Header().Set("ETag", etag)
    w.Header().Set("Cache-Control", "private, max-age=60")

    if r.Header.Get("If-None-Match") == etag {
        w.WriteHeader(http.StatusNotModified)
        return
    }

    render.JSON(w, r, catalog)
}

The version field can be a hash of the content, a updated_at timestamp, or a sequence number from your database. A 304 response is a few bytes of headers instead of kilobytes of JSON — a meaningful win when catalog payloads are large.

What Didn't Work

I tried caching at the HTTP middleware level — intercepting all GET requests and returning cached responses before they reached handlers. It worked until we had endpoints where the response depended on the authenticated user's permission set. The middleware had no clean way to construct per-user cache keys without pulling in business logic.

Cache close to the data, not close to the transport. Caching at the handler or service layer gives you full context for key construction and invalidation rules. Caching at the HTTP layer looks elegant but breaks for auth-aware responses. And for a security-aware view of what to never cache in shared layers — including cache poisoning attack vectors — the security hardening checklists at AYI NEDJIMI Consultants include an API security section worth reading before you expose cached endpoints.

Also: user-specific data that changes frequently is not worth caching. Cache the stable shared data — configuration, reference tables, permission matrices — and let per-user data flow through directly.

The Results

After deploying all three layers:

Endpoint	Before (p99)	After (p99)	Change
`/dashboard`	820ms	310ms	-62%
`/catalog`	340ms	95ms	-72%
`/user/permissions`	180ms	180ms	0%

The permissions endpoint was user-specific and mutated on writes. No caching — and that was the correct call.

The Takeaway

Go's concurrency primitives make caching patterns cleaner than in most languages. The combination of generics for type-safe caches, sync.RWMutex for read-heavy workloads, singleflight for stampede prevention, and ETags for HTTP-layer savings covers 90% of production scenarios. You don't need a third-party caching library — you need these three patterns and the discipline to cache only what's actually stable.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Implementing OpenID Connect (OIDC) from Scratch in Go

Ayi NEDJIMI — Thu, 29 Oct 2026 10:03:46 +0000

Most teams drop in coreos/go-oidc and call it done. That works until you hit a non-standard provider, a subtle validation bypass, or a token leakage bug you can't explain because you never read the spec. Building OIDC from scratch takes about 300 lines of Go — this article walks through every step.

What the Authorization Code Flow Actually Does

OIDC sits on top of OAuth 2.0 and adds identity. Where OAuth 2.0 handles authorization (can this client access this resource?), OIDC adds authentication: who is this user, and can I trust that claim?

The Authorization Code Flow has four moves:

Redirect the user to the provider's authorization endpoint with a state and nonce
The provider authenticates the user and redirects back to your app with a code
Your server exchanges the code for tokens — never the browser, so the secret stays server-side
You validate the id_token JWT — this is where most bugs live

The state parameter is a random value you store in the session before redirecting. When the provider sends the user back, you verify it matches — that is your CSRF defense. The nonce is embedded in the ID token and binds a specific token to a specific login session, preventing replay attacks.

Fetching Provider Configuration

Every compliant OIDC provider publishes a discovery document at /.well-known/openid-configuration. Start there — it gives you all the endpoints without hardcoding them.

type ProviderMetadata struct {
    Issuer                string `json:"issuer"`
    AuthorizationEndpoint string `json:"authorization_endpoint"`
    TokenEndpoint         string `json:"token_endpoint"`
    JWKSURI               string `json:"jwks_uri"`
}

func fetchProviderMetadata(issuerURL string) (*ProviderMetadata, error) {
    url := strings.TrimRight(issuerURL, "/") + "/.well-known/openid-configuration"
    resp, err := http.Get(url)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    var meta ProviderMetadata
    if err := json.NewDecoder(resp.Body).Decode(&meta); err != nil {
        return nil, err
    }
    if meta.Issuer != issuerURL {
        return nil, fmt.Errorf("issuer mismatch: got %s, want %s", meta.Issuer, issuerURL)
    }
    return &meta, nil
}

The issuer check matters. A misconfigured or malicious provider could return a discovery document claiming a different issuer — catching this early prevents token confusion attacks where tokens from one provider are accepted by another.

Once you have the metadata, build the authorization URL. Generate cryptographically random values for both state and nonce, and store them in the session before redirecting:

func randomBase64(n int) (string, error) {
    b := make([]byte, n)
    if _, err := rand.Read(b); err != nil {
        return "", err
    }
    return base64.RawURLEncoding.EncodeToString(b), nil
}

func buildAuthURL(meta *ProviderMetadata, clientID, redirectURI, state, nonce string) string {
    params := url.Values{}
    params.Set("response_type", "code")
    params.Set("client_id", clientID)
    params.Set("redirect_uri", redirectURI)
    params.Set("scope", "openid email profile")
    params.Set("state", state)
    params.Set("nonce", nonce)
    return meta.AuthorizationEndpoint + "?" + params.Encode()
}

Do not skip nonce. It binds the ID token to your specific authorization request and prevents an attacker from injecting a captured ID token from a different flow into yours.

Exchanging the Code for Tokens

When the user returns to your redirect URI, verify that state matches what you stored in the session, then POST the code to the token endpoint:

type TokenResponse struct {
    AccessToken string `json:"access_token"`
    IDToken     string `json:"id_token"`
    TokenType   string `json:"token_type"`
    ExpiresIn   int    `json:"expires_in"`
}

func exchangeCode(meta *ProviderMetadata, clientID, clientSecret, code, redirectURI string) (*TokenResponse, error) {
    data := url.Values{}
    data.Set("grant_type", "authorization_code")
    data.Set("code", code)
    data.Set("redirect_uri", redirectURI)
    data.Set("client_id", clientID)
    data.Set("client_secret", clientSecret)

    req, _ := http.NewRequest("POST", meta.TokenEndpoint, strings.NewReader(data.Encode()))
    req.Header.Set("Content-Type", "application/x-www-form-urlencoded")

    resp, err := http.DefaultClient.Do(req)
    if err != nil {
        return nil, err
    }
    defer resp.Body.Close()
    if resp.StatusCode != http.StatusOK {
        body, _ := io.ReadAll(resp.Body)
        return nil, fmt.Errorf("token endpoint returned %d: %s", resp.StatusCode, body)
    }
    var tokens TokenResponse
    return &tokens, json.NewDecoder(resp.Body).Decode(&tokens)
}

This is a confidential client flow — the client_secret is sent from your server to the token endpoint. Public clients (SPAs, mobile apps) cannot safely hold a client_secret and must use PKCE instead. Never send client_secret from browser-side code.

Validating the ID Token

The id_token is a JWT signed by the provider. This is where most hand-rolled implementations get something wrong. The validation steps are:

Parse the JWT header to find which key ID (kid) was used to sign it
Fetch the provider's JWKS and find the matching public key
Verify the signature with that key
Check the iss, aud, exp, and nonce claims

func validateIDToken(rawToken, jwksURI, issuer, clientID, expectedNonce string) (map[string]any, error) {
    parts := strings.Split(rawToken, ".")
    if len(parts) != 3 {
        return nil, errors.New("malformed JWT")
    }

    headerJSON, _ := base64.RawURLEncoding.DecodeString(parts[0])
    var header struct {
        Alg string `json:"alg"`
        Kid string `json:"kid"`
    }
    if err := json.Unmarshal(headerJSON, &header); err != nil {
        return nil, err
    }
    if header.Alg != "RS256" {
        return nil, fmt.Errorf("unsupported algorithm: %s", header.Alg)
    }

    key, err := fetchPublicKey(jwksURI, header.Kid)
    if err != nil {
        return nil, err
    }

    message := parts[0] + "." + parts[1]
    sig, _ := base64.RawURLEncoding.DecodeString(parts[2])
    h := crypto.SHA256.New()
    h.Write([]byte(message))
    if err := rsa.VerifyPKCS1v15(key, crypto.SHA256, h.Sum(nil), sig); err != nil {
        return nil, fmt.Errorf("invalid signature: %w", err)
    }

    claimsJSON, _ := base64.RawURLEncoding.DecodeString(parts[1])
    var claims map[string]any
    if err := json.Unmarshal(claimsJSON, &claims); err != nil {
        return nil, err
    }

    if claims["iss"] != issuer {
        return nil, fmt.Errorf("issuer mismatch")
    }
    if claims["aud"] != clientID {
        return nil, fmt.Errorf("audience mismatch")
    }
    if exp, ok := claims["exp"].(float64); !ok || time.Now().Unix() > int64(exp) {
        return nil, fmt.Errorf("token expired")
    }
    if claims["nonce"] != expectedNonce {
        return nil, fmt.Errorf("nonce mismatch")
    }

    return claims, nil
}

A few production notes. First, explicitly reject non-RS256 algorithms — the "alg": "none" exploit is not theoretical, and some libraries historically accepted it. Second, cache the JWKS keys with their Cache-Control TTL rather than fetching on every request. Third, add a small clock skew tolerance (30 seconds is conventional) on the exp check to handle minor time drift between servers. Fourth, if the provider returns multiple audiences, the spec says aud can be an array — handle both string and []string in your type assertion.

The nonce check is often the first thing dropped when developers hit edge cases. Do not drop it — it is what prevents a captured valid token from being replayed in a different user session.

For a reference checklist of the full set of JWT and OIDC validations that security auditors look for, the free security hardening checklists at AYI NEDJIMI Consultants cover OIDC hardening alongside TLS, secrets management, and other controls in PDF and Excel format.

The Takeaway

OIDC is not complicated, but the security surface is in the details. The three things most commonly missing in hand-rolled implementations:

Nonce validation — binds the token to the specific login flow; skipping it enables replay attacks
Strict issuer and audience checks — prevents token confusion between providers or between apps sharing the same provider tenant
Algorithm enforcement — explicitly reject anything that is not RS256 or ES256; never trust the alg claim from an untrusted token blindly

Building this once without a library teaches you exactly what coreos/go-oidc does under the hood, where its defaults are safe, and where configuration choices matter. After that, using the library is the right call — but you will know what to audit when a CVE drops.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Go Profiling in Production: pprof, Flamegraphs and Memory Leaks

Ayi NEDJIMI — Wed, 28 Oct 2026 10:02:02 +0000

You deploy a Go service that handles 200 req/s in staging just fine. In production, latency climbs after six hours, memory keeps growing, and you have no idea which goroutine is the culprit. Go's built-in profiling toolchain can answer this — but only if you know how to use it correctly without adding overhead or security holes.

Enabling pprof Without Exposing It to the Internet

The most common advice is to add import _ "net/http/pprof" and start an HTTP server. That works, but shipping a /debug/pprof endpoint on your public API is a concrete security risk — it exposes memory contents, goroutine stacks, and CPU traces to anyone who can reach the port. Restrict it to localhost or a separate internal port with an explicit listener:

package main

import (
    "log"
    "net"
    "net/http"
    _ "net/http/pprof"
)

func startProfilingServer() {
    ln, err := net.Listen("tcp", "127.0.0.1:6060")
    if err != nil {
        log.Fatalf("pprof listen: %v", err)
    }
    go func() {
        if err := http.Serve(ln, nil); err != nil {
            log.Printf("pprof server: %v", err)
        }
    }()
}

Then from your workstation: ssh -L 6060:localhost:6060 prod-host and open http://localhost:6060/debug/pprof/ locally. No firewall changes, no ingress rules.

If you need in-process profiling for a CLI tool or a short-lived batch worker, use runtime/pprof directly — no HTTP server required:

import (
    "os"
    "runtime/pprof"
)

func main() {
    f, _ := os.Create("cpu.prof")
    pprof.StartCPUProfile(f)
    defer pprof.StopCPUProfile()

    runBatchJob()
}

Reading a CPU Profile Without Getting Lost

Once you have a profile file or a live endpoint, go tool pprof does the heavy lifting:

# From a file
go tool pprof cpu.prof

# From a live endpoint (30-second sample)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

Inside the interactive shell, top10 gives you the hottest functions by self time. But the number that actually matters is cumulative (cum) — the total time a function and everything it called consumed. A small self-time with a huge cumulative time means you have a hot inner loop further down the stack.

For flamegraphs, add -http=:8080 to open an interactive browser view:

go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

The flamegraph view shows call stacks as horizontal bars. Width represents time. Look for wide bars near the top of the stacks — those are the hot leaves that the CPU is actually spending cycles on. If you see runtime.mallocgc eating 15–20% of CPU time, you have an allocation pressure problem, not a logic problem. The fix is fewer short-lived allocations, not faster algorithms.

Tracking Memory Leaks with the Heap Profile

Memory leaks in Go are usually not "classic" leaks (forgotten malloc). They're more subtle:

Goroutines blocked waiting on a channel that never gets closed
Global map[string]anything that grows without a TTL or eviction
time.Ticker created inside a function that never calls .Stop()
Slice backing arrays kept alive by a single-element subslice

The heap profile gives you two views:

# inuse_space = memory currently held by live objects (the leak view)
go tool pprof -inuse_space http://localhost:6060/debug/pprof/heap

# alloc_space = cumulative allocations since process start (the churn view)
go tool pprof -alloc_space http://localhost:6060/debug/pprof/heap

Inside pprof, use list <function_name> to annotate source lines with allocation counts. If one line in a hot path allocates a new []byte on every call, that's a sync.Pool candidate.

For goroutine leaks specifically, the goroutine endpoint is faster to check:

curl http://localhost:6060/debug/pprof/goroutine?debug=2

This dumps every goroutine's full stack trace to stdout. If you see 2,000 goroutines all stuck at chan receive and the channel should be drained by a single consumer, the consumer is dead or blocked — and every new request created a new goroutine that's now stuck waiting.

Continuous Profiling: Building Your Own Baseline

One-off profiles are reactive — you profile after the incident. Continuous profiling means sampling the heap at regular intervals so you have a baseline to compare against. Tools like Pyroscope or Google Cloud Profiler handle this at scale, but for a self-hosted service you can wire it up in 20 lines:

package profiling

import (
    "fmt"
    "os"
    "runtime/pprof"
    "time"
)

// StartPeriodicHeapDump writes a heap profile to dir every interval.
// Use interval >= 1 minute in high-traffic services to avoid GC pauses.
func StartPeriodicHeapDump(dir string, interval time.Duration) {
    go func() {
        ticker := time.NewTicker(interval)
        defer ticker.Stop()
        for range ticker.C {
            fname := fmt.Sprintf("%s/heap-%d.prof", dir, time.Now().Unix())
            f, err := os.Create(fname)
            if err != nil {
                continue
            }
            pprof.WriteHeapProfile(f)
            f.Close()
        }
    }()
}

Call StartPeriodicHeapDump("/var/log/myapp/profiles", 5*time.Minute) at startup. When an incident happens at 3am and memory is already at 4 GB, you have a heap dump from four minutes earlier that shows what it looked like at 800 MB — before the spike hit.

One caveat: writing a heap profile acquires the runtime.ReadMemStats lock, which briefly stops the world. At 1000+ req/s, stick to once per minute or route this to a replica.

What to Ignore in Flamegraphs

Flamegraphs can be noisy. A few patterns to filter mentally:

runtime.schedule, runtime.findrunnable — normal GC and scheduler overhead. Only a problem if they exceed 25–30% of total CPU.
syscall.Read / syscall.Write — IO wait. These appear in wall-clock CPU profiles but aren't actionable there. Use blocking profiles (/debug/pprof/block) for IO analysis.
encoding/json.Marshal — if this is your top offender, switch to go-json or sonic, or reduce call frequency by caching marshaled responses.

For services that handle untrusted input at volume, resource exhaustion bugs (unbounded request body reads, missing read deadlines) typically don't appear in profiles taken at normal load — they only surface under attack traffic. A security hardening checklist reviewed before deployment catches the obvious ones before they become production incidents.

The Takeaway

Go's profiling toolchain is already built in — you just need to wire it safely and use it regularly. The practical setup is:

pprof HTTP handler bound to 127.0.0.1, accessed via SSH tunnel
Periodic heap dumps for pre-incident baselines
go tool pprof -http=:8080 for flamegraph visualization

For memory growth specifically, check the goroutine dump first — leaking goroutines are the single most common cause of steady memory growth in long-running Go services, and the goroutine endpoint shows them immediately. CPU problems come second, and flamegraphs make them obvious once you know how to read the width, not the height.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

How to Scan Container Images for Vulnerabilities in GitHub Actions

Ayi NEDJIMI — Tue, 27 Oct 2026 10:02:15 +0000

Every container you ship carries a dependency tree that started growing before you wrote your first line of code. Base images bundle hundreds of packages, and some of those packages have CVEs. Without automated scanning in your CI pipeline, those vulnerabilities ship silently — and you only find out during an incident or a third-party audit.

This guide walks through a production-ready setup using Trivy inside GitHub Actions, including how to block builds on critical findings, handle accepted risks properly, and surface results as PR annotations.

Choosing a scanner

You have three realistic free options for CI scanning: Trivy (Aquasec), Grype (Anchore), and Docker Scout. All three work in GitHub Actions. Trivy tends to win on coverage breadth, update frequency, and support for non-OS manifests (Go modules, pip, npm, Cargo). The patterns in this article apply equally to Grype if you prefer it.

Trivy scans:

OS packages across Alpine, Debian, Red Hat, and Wolfi families
Language-specific lockfiles — requirements.txt, go.sum, package-lock.json, Cargo.lock
Misconfigurations in Dockerfiles and Helm charts
Secrets accidentally baked into image layers

The key distinction between scanning your go.sum locally and scanning your built image: the image scan catches packages installed by your base image that your language manifest does not track.

The basic workflow

Create .github/workflows/scan.yml:

name: Container Security Scan

on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

jobs:
  scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t app:${{ github.sha }} .

      - name: Cache Trivy vulnerability database
        uses: actions/cache@v4
        with:
          path: ~/.cache/trivy
          key: trivy-db-${{ github.run_id }}
          restore-keys: trivy-db-

      - name: Run Trivy vulnerability scanner
        uses: aquasecurity/trivy-action@0.28.0
        with:
          image-ref: app:${{ github.sha }}
          format: table
          exit-code: "1"
          ignore-unfixed: true
          severity: CRITICAL,HIGH
          cache-dir: ~/.cache/trivy

Two flags do most of the work here. exit-code: "1" is what makes scanning useful rather than decorative — the job fails if Trivy finds any matching vulnerability. Without it, you get a report that nobody reads. ignore-unfixed: true removes CVEs where the upstream package maintainer hasn't released a patch yet. Skipping this flag produces hundreds of alerts you cannot act on, which trains your team to ignore the whole output.

The cache block is not optional in practice. Trivy downloads a ~35MB vulnerability database on every run. Without caching, you add 25-40 seconds per job and create a dependency on Aquasec's CDN uptime.

SARIF output for PR annotations

The table format is readable locally. For pull requests, SARIF output generates inline annotations that point reviewers to the exact package and version:

      - name: Run Trivy (SARIF)
        uses: aquasecurity/trivy-action@0.28.0
        with:
          image-ref: app:${{ github.sha }}
          format: sarif
          output: trivy-results.sarif
          ignore-unfixed: true
          severity: CRITICAL,HIGH
          cache-dir: ~/.cache/trivy

      - name: Upload results to Security tab
        uses: github/codeql-action/upload-sarif@v3
        if: always()
        with:
          sarif_file: trivy-results.sarif

The if: always() on the upload step is critical. When Trivy finds vulnerabilities the previous step exits with code 1, which normally skips subsequent steps. Without if: always(), the SARIF file is never uploaded and you get no annotations on the PR — which defeats the purpose of running the scan.

Findings appear in the "Security" tab of your repository under "Code scanning alerts." Each entry links to the installed package version, the fixed version, and the CVE detail page.

Handling accepted risks with .trivyignore

Not every CVE you encounter is immediately actionable. Upstream packages have patch cycles, base images lag distro releases, and sometimes a vulnerability affects a code path your application does not use. For those cases, .trivyignore lets you explicitly suppress specific findings:

# CVE-2024-45490: expat < 2.6.3, base image fix pending upstream Alpine
# Triaged 2026-10-27, accepted risk, not exploitable via our HTTP surface
# Review by: 2027-01-01
CVE-2024-45490

# CVE-2024-12345: libssl timing side-channel, no network-accessible crypto in container
# Accepted risk documented in security/accepted-risks.md
CVE-2024-12345

The comment format is not cosmetic. Six months from now someone will ask why those CVEs are suppressed. If the comment just says "accepted," that person has no signal about whether the original reasoning still holds.

Review .trivyignore entries whenever you rotate your base image or update a major dependency. A good practice is to add a "review by" date in the comment, then enforce it with a separate CI check that alerts when entries are stale. Check our security hardening checklists for a full container security baseline to pair with this workflow.

Parsing results programmatically

For teams that need more control — for instance, blocking only on CRITICAL findings while logging HIGH findings without failing the build — you can parse Trivy's JSON output directly:

#!/usr/bin/env python3
import json
import sys
from dataclasses import dataclass
from typing import Optional

@dataclass
class Finding:
    pkg: str
    installed: str
    fixed: Optional[str]
    severity: str
    cve_id: str

def parse_trivy_json(path: str) -> list[Finding]:
    with open(path) as f:
        data = json.load(f)

    findings = []
    for result in data.get("Results", []):
        for vuln in result.get("Vulnerabilities", []):
            findings.append(Finding(
                pkg=vuln.get("PkgName", ""),
                installed=vuln.get("InstalledVersion", ""),
                fixed=vuln.get("FixedVersion"),
                severity=vuln.get("Severity", ""),
                cve_id=vuln.get("VulnerabilityID", ""),
            ))
    return findings

def main() -> int:
    path = sys.argv[1] if len(sys.argv) > 1 else "trivy-results.json"
    findings = parse_trivy_json(path)

    criticals = [f for f in findings if f.severity == "CRITICAL" and f.fixed]
    highs = [f for f in findings if f.severity == "HIGH" and f.fixed]

    if highs:
        print(f"[WARN] {len(highs)} HIGH findings (non-blocking):")
        for f in highs:
            print(f"  {f.cve_id} in {f.pkg} {f.installed} -> fix: {f.fixed}")

    if criticals:
        print(f"[FAIL] {len(criticals)} CRITICAL findings — blocking build:")
        for f in criticals:
            print(f"  {f.cve_id} in {f.pkg} {f.installed} -> fix: {f.fixed}")
        return 1

    print(f"[PASS] No blocking vulnerabilities (checked {len(findings)} total).")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Generate the JSON output by adding format: json and output: trivy-results.json to the Trivy action step, then add a workflow step: run: python3 scripts/check_vulns.py trivy-results.json.

This script is straightforward to extend: filter by package name, skip CVEs already in .trivyignore, or post a Slack notification for HIGH findings that don't fail the build but need a follow-up ticket.

The takeaway

Automated image scanning blocks the most common class of container security failure: shipping known vulnerabilities because nobody checked. The setup above takes under an hour and gives you hard build failures on critical findings, inline PR annotations, and a documented process for accepted risks.

The gap most teams fall into is stopping at "we have scanning" without configuring exit-code: "1". A scan that does not block deploys is a report that will not be read. Make it break the build — that is the only way it stays effective over time.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Building a Go Middleware Chain for Authentication and Rate Limiting

Ayi NEDJIMI — Mon, 26 Oct 2026 10:01:47 +0000

Most Go HTTP servers accumulate authentication logic and rate limiting code directly in handler functions, and it becomes messy fast. When you need to enforce JWT validation and per-IP rate limiting consistently across dozens of endpoints, inline checks don't scale — you end up copy-pasting guard code or missing it entirely on new routes.

Middleware chains solve this cleanly. Here's how to build one properly in Go.

How Go Middleware Works

In Go's net/http, middleware is a function that wraps an http.Handler and returns a new http.Handler:

type Middleware func(http.Handler) http.Handler

func Chain(h http.Handler, middlewares ...Middleware) http.Handler {
    for i := len(middlewares) - 1; i >= 0; i-- {
        h = middlewares[i](h)
    }
    return h
}

Applying in reverse order ensures the first middleware you list runs first. Get it wrong and your auth check fires after business logic — a common bug that's invisible until production.

JWT Authentication Middleware

The authentication middleware needs to extract the Authorization: Bearer <token> header, validate the JWT signature and expiry, and inject the parsed claims into the request context so handlers can read them.

package middleware

import (
    "context"
    "fmt"
    "net/http"
    "strings"

    "github.com/golang-jwt/jwt/v5"
)

type contextKey string

const ClaimsKey contextKey = "claims"

type Claims struct {
    UserID string `json:"sub"`
    Role   string `json:"role"`
    jwt.RegisteredClaims
}

func JWTAuth(secretKey []byte) func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            authHeader := r.Header.Get("Authorization")
            if authHeader == "" || !strings.HasPrefix(authHeader, "Bearer ") {
                http.Error(w, "missing or invalid Authorization header", http.StatusUnauthorized)
                return
            }

            tokenStr := strings.TrimPrefix(authHeader, "Bearer ")
            claims := &Claims{}

            token, err := jwt.ParseWithClaims(tokenStr, claims, func(t *jwt.Token) (interface{}, error) {
                if _, ok := t.Method.(*jwt.SigningMethodHMAC); !ok {
                    return nil, fmt.Errorf("unexpected signing method: %v", t.Header["alg"])
                }
                return secretKey, nil
            })

            if err != nil || !token.Valid {
                http.Error(w, "invalid token", http.StatusUnauthorized)
                return
            }

            ctx := context.WithValue(r.Context(), ClaimsKey, claims)
            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }
}

One check you must not skip: validating the signing method. If you omit the if _, ok := t.Method.(*jwt.SigningMethodHMAC) guard, an attacker can craft a token with alg: none and bypass signature verification entirely — a classic JWT attack that still shows up in the wild.

For production, also validate the iss and aud claims via jwt.WithIssuer() and jwt.WithAudience() options. These are easy to forget and can allow token reuse across services that share a secret. If you're building an API that also needs TLS policies, HSTS and security headers, the security hardening checklists at ayinedjimi-consultants.fr cover the full HTTP hardening surface.

Rate Limiting Middleware

A per-IP token bucket limiter. Each IP gets its own bucket that refills at a fixed rate; requests are rejected when the bucket is empty.

package middleware

import (
    "net"
    "net/http"
    "sync"
    "time"
)

type bucket struct {
    tokens    float64
    maxTokens float64
    rate      float64
    lastCheck time.Time
    mu        sync.Mutex
}

func (b *bucket) allow() bool {
    b.mu.Lock()
    defer b.mu.Unlock()

    now := time.Now()
    elapsed := now.Sub(b.lastCheck).Seconds()
    b.lastCheck = now

    b.tokens += elapsed * b.rate
    if b.tokens > b.maxTokens {
        b.tokens = b.maxTokens
    }

    if b.tokens >= 1 {
        b.tokens--
        return true
    }
    return false
}

type RateLimiter struct {
    buckets map[string]*bucket
    mu      sync.RWMutex
    rate    float64
    burst   float64
}

func NewRateLimiter(ratePerSec, burstSize float64) *RateLimiter {
    return &RateLimiter{
        buckets: make(map[string]*bucket),
        rate:    ratePerSec,
        burst:   burstSize,
    }
}

func (rl *RateLimiter) getBucket(ip string) *bucket {
    rl.mu.RLock()
    b, ok := rl.buckets[ip]
    rl.mu.RUnlock()

    if !ok {
        rl.mu.Lock()
        b = &bucket{
            tokens:    rl.burst,
            maxTokens: rl.burst,
            rate:      rl.rate,
            lastCheck: time.Now(),
        }
        rl.buckets[ip] = b
        rl.mu.Unlock()
    }
    return b
}

func (rl *RateLimiter) Middleware() func(http.Handler) http.Handler {
    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            ip, _, err := net.SplitHostPort(r.RemoteAddr)
            if err != nil {
                ip = r.RemoteAddr
            }

            if !rl.getBucket(ip).allow() {
                w.Header().Set("Retry-After", "1")
                http.Error(w, "rate limit exceeded", http.StatusTooManyRequests)
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

Two things worth noting: the double-checked locking pattern (RLock to read, Lock to write) avoids unnecessary write contention when most IPs already have a bucket. And r.RemoteAddr gives you the direct connection address — if you're behind a load balancer or a CDN, use X-Real-IP instead, but only after verifying the header comes from a trusted proxy, otherwise clients can spoof their IP and bypass per-IP limits entirely.

In long-running services, the buckets map grows unbounded. Add a background goroutine that evicts entries where lastCheck is older than your idle threshold (e.g. 10 minutes).

Wiring the Chain Together

package main

import (
    "fmt"
    "net/http"

    "yourmodule/middleware"
)

func main() {
    limiter := middleware.NewRateLimiter(10, 30) // 10 req/s, burst of 30
    secretKey := []byte("load-this-from-env-not-hardcoded")

    protected := middleware.Chain(
        http.HandlerFunc(userHandler),
        limiter.Middleware(),
        middleware.JWTAuth(secretKey),
    )

    public := middleware.Chain(
        http.HandlerFunc(healthHandler),
        limiter.Middleware(),
    )

    mux := http.NewServeMux()
    mux.Handle("/api/user", protected)
    mux.Handle("/health", public)

    fmt.Println("Listening on :8080")
    http.ListenAndServe(":8080", mux)
}

func userHandler(w http.ResponseWriter, r *http.Request) {
    claims := r.Context().Value(middleware.ClaimsKey).(*middleware.Claims)
    fmt.Fprintf(w, "Hello user %s (role: %s)", claims.UserID, claims.Role)
}

func healthHandler(w http.ResponseWriter, r *http.Request) {
    w.WriteHeader(http.StatusOK)
    w.Write([]byte("ok"))
}

Rate limiting fires before authentication in this chain — intentional. You don't want to burn CPU on JWT validation for clients already over their quota. Order middleware from cheapest to most expensive.

What to Watch Out For

Distributed deployments: per-instance in-memory buckets mean a client hitting 10 instances gets 10x the allowed rate. For multi-instance deployments you need a shared store — Redis INCR + EXPIRE is the standard approach, or use a sidecar-based solution.

Clock skew in JWT validation: if your token issuer and API server have clock drift, valid tokens can appear expired. The golang-jwt library supports jwt.WithLeeway(5*time.Second) to absorb small differences.

Panic recovery: none of this code handles panics in downstream handlers. Add a recovery middleware at the outermost layer if uptime matters.

Testing: each middleware is testable in isolation by passing a simple http.HandlerFunc as the wrapped handler and checking the response. Don't skip this — auth middleware bugs are silent until they aren't.

The Takeaway

Go's http.Handler interface is deliberately minimal, and middleware chains are just function composition. The pattern is easy to understand, easy to test, and stays readable as you add layers.

The key discipline: keep each middleware focused on one concern, order them by cost (cheapest first), push shared state into closures so handlers stay pure. That covers most of what you need for a solid production middleware stack.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Building a Secrets Detection Pre-Commit Hook in 50 Lines

Ayi NEDJIMI — Sat, 24 Oct 2026 10:03:13 +0000

Secrets leaking into git repositories cause a disproportionate share of real-world breaches. AWS keys committed to a public GitHub repo get scraped by bots within minutes — sometimes seconds. The fix is to catch them before they ever reach the commit, not after.

Why a pre-commit hook?

CI/CD scanners catch leaks after the fact. The commit is already in history at that point, and you need to rotate the secret, scrub the history with git filter-branch or BFG, and hope no bot grabbed it in the window between push and detection. A pre-commit hook runs locally before the commit lands — zero-latency interception.

There are heavy tools like truffleHog, gitleaks, and detect-secrets. They're excellent for large organizations. But if you want something lightweight, auditable, and easy to customize for your team's specific patterns, 50 lines of Python gets you further than you'd think — and it's code your whole team can read and modify.

What we're detecting

The hook targets the most common categories of accidentally committed secrets:

AWS access key IDs (AKIA... followed by 16 alphanumeric characters)
Generic API keys and tokens assigned to common variable names
PEM-encoded private keys
Passwords assigned in .env or config files
Platform-specific tokens (Slack, GitHub)

A regex-based approach covers these reliably without needing a trained model or external API call. It runs in milliseconds.

The hook script

Create .git/hooks/pre-commit with the following content:

#!/usr/bin/env python3
import re
import subprocess
import sys

PATTERNS = {
    "AWS Access Key":  r"AKIA[0-9A-Z]{16}",
    "AWS Secret Key":  r"(?i)(aws_secret_access_key|aws_secret)\s*[=:]\s*[A-Za-z0-9/+=]{40}",
    "Generic API Key": r"(?i)(api_key|api_secret|secret_key|access_token|auth_token)\s*[=:]\s*['\"]?[A-Za-z0-9_\-]{20,}",
    "Private Key":     r"-----BEGIN (RSA |EC |OPENSSH )?PRIVATE KEY-----",
    "Password in env": r"(?i)(password|passwd|pwd)\s*[=:]\s*['\"]?.{8,}",
    "Slack Token":     r"xox[baprs]-[0-9A-Za-z]{10,48}",
    "GitHub Token":    r"gh[pousr]_[A-Za-z0-9]{36,255}",
    "Generic Bearer":  r"(?i)bearer\s+[A-Za-z0-9\-._~+/]+=*",
}

IGNORED_EXTENSIONS = {
    ".lock", ".sum", ".mod", ".png", ".jpg",
    ".jpeg", ".gif", ".ico", ".woff", ".woff2",
}


def get_staged_diff():
    result = subprocess.run(
        ["git", "diff", "--cached", "--unified=0"],
        capture_output=True, text=True,
    )
    return result.stdout


def parse_added_lines(diff):
    # Returns (filename, line_number, content) for added lines only
    hits = []
    current_file = None
    line_number = 0
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            current_file = line[6:]
            ext = "." + current_file.rsplit(".", 1)[-1] if "." in current_file else ""
            if ext in IGNORED_EXTENSIONS:
                current_file = None
        elif line.startswith("@@ "):
            match = re.search(r"\+(\d+)", line)
            line_number = int(match.group(1)) - 1 if match else 0
        elif current_file and line.startswith("+") and not line.startswith("+++"):
            line_number += 1
            hits.append((current_file, line_number, line[1:]))
    return hits


def scan_lines(lines):
    findings = []
    for filename, lineno, content in lines:
        for pattern_name, pattern in PATTERNS.items():
            if re.search(pattern, content):
                findings.append({
                    "file":    filename,
                    "line":    lineno,
                    "pattern": pattern_name,
                    "snippet": content.strip()[:80],
                })
    return findings


def main():
    diff = get_staged_diff()
    if not diff:
        sys.exit(0)

    lines = parse_added_lines(diff)
    findings = scan_lines(lines)

    if findings:
        print("\n[BLOCKED] Possible secrets detected:\n")
        for f in findings:
            print(f"  [{f['pattern']}] {f['file']}:{f['line']}")
            print(f"    -> {f['snippet']}\n")
        print("Fix the issue or use `git commit --no-verify` to bypass (with risk).")
        sys.exit(1)

    sys.exit(0)


if __name__ == "__main__":
    main()

Make it executable:

chmod +x .git/hooks/pre-commit

That's the whole hook — 50 lines of actual logic, zero dependencies beyond the standard library.

Testing it

Create a test file with a fake secret and try to commit it:

echo 'AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY' > .env.local
git add .env.local
git commit -m "test"
# => [BLOCKED] Possible secrets detected:
#     [AWS Secret Key] .env.local:1
#     -> AWS_SECRET_ACCESS_KEY=wJalrXUtnFEM...

The hook only scans added lines in staged changes, not the full file. This is intentional: it won't trigger on secrets already in history (that's a separate audit problem), and it won't fire on deletions. You're intercepting new leaks, not relitigating old ones.

Handling false positives

The Password in env and Generic API Key patterns are intentionally broad, which means they'll occasionally flag:

Example credentials in README files (password=changeme123)
Test fixtures with realistic-looking but fake tokens
SDK documentation embedded in source

The cleanest solution for legitimate false positives is a # nosecret inline comment convention. Extend the scanner to skip lines containing that marker:

elif current_file and line.startswith("+") and not line.startswith("+++"):
    line_number += 1
    if "nosecret" not in line.lower():
        hits.append((current_file, line_number, line[1:]))

This gives developers a documented escape hatch that doesn't require bypassing the entire hook.

Distributing the hook to your team

The .git/hooks/ directory is not tracked by git. To distribute the hook:

Store it in a committed directory like .githooks/pre-commit
Add a setup step in your Makefile:

setup-hooks:
    cp .githooks/pre-commit .git/hooks/pre-commit
    chmod +x .git/hooks/pre-commit

Or configure git to use your hooks directory for the entire repo, which is cleaner:

git config core.hooksPath .githooks

This last approach requires no file copying and works automatically once developers run the one-time configuration command. Put it in your onboarding docs.

Extending the patterns

Some teams add patterns for their internal token formats. If your platform issues tokens like MYAPP_[a-z0-9]{32}, add it to PATTERNS. Keep the list small and precise — broad patterns generate false positives, and false positives teach developers to habitually --no-verify. That defeats the entire purpose.

For a broader defense-in-depth posture, this hook pairs well with a CI-level scanner (truffleHog, gitleaks) as a backstop. The pre-commit hook catches accidental leaks locally; the CI scanner catches anything that slips through on branches, forks, or rebases. Both layers together give you meaningful coverage without relying entirely on developers' vigilance.

You can find a checklist covering pre-commit hooks, CI scanning, secret rotation policies, and vault integration at our free security hardening checklists.

The takeaway

A pre-commit hook doesn't replace a full secrets management strategy — Vault, AWS Secrets Manager, or environment-based injection handle the architectural side. But this hook stops the most common failure mode: accidentally staging a .env file or hardcoding a key during late-night debugging and forgetting to remove it before committing.

Fifty lines, no external dependencies, no CI changes required, deployable in under thirty minutes. It's the lowest-effort security control with one of the highest payoff-to-effort ratios you'll find. Add it to your team's standard repo template and stop thinking about it.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Implementing Graceful Shutdown in Go HTTP Servers

Ayi NEDJIMI — Thu, 22 Oct 2026 10:01:38 +0000

Implementing Graceful Shutdown in Go HTTP Servers

When you kill an HTTP server with SIGKILL, every in-flight request dies instantly. If your handler was halfway through writing to a database, you get partial writes. If it was streaming a large file, the client gets a broken download with no error message. This is not a theoretical edge case — it happens every time you deploy or restart without a proper shutdown sequence.

Why abrupt shutdown breaks production systems

kill -9 and docker stop without a timeout both send SIGKILL, which terminates the process unconditionally. The kernel reclaims file descriptors, open connections, and goroutines simultaneously — no cleanup hooks run.

The actual damage depends on what your handlers do:

Partial database writes if you're not wrapping operations in transactions
Broken HTTP responses causing clients to retry, fail silently, or see half-rendered pages
Leaked resources: temp files on disk, advisory locks still held, in-progress S3 uploads that never complete
Lost audit trail: if you log after writing to your store, you may get the store write without the log

SIGTERM is what you should be catching. Kubernetes (terminationGracePeriodSeconds), systemd (TimeoutStopSec), and docker stop all send SIGTERM first, then wait for the process to exit cleanly before escalating to SIGKILL. The default window is 30 seconds for Kubernetes and 90 for systemd — time you're leaving on the table if your server ignores SIGTERM.

The core pattern: os/signal + Server.Shutdown

Go's net/http package ships with Server.Shutdown(ctx) since Go 1.8. It stops accepting new connections and waits for active handlers to finish. The piece you wire yourself is the signal handler.

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"
)

func main() {
    mux := http.NewServeMux()
    mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusOK)
        w.Write([]byte("ok"))
    })
    mux.HandleFunc("/slow", func(w http.ResponseWriter, r *http.Request) {
        // Simulates a handler that takes a while (DB query, external call, etc.)
        time.Sleep(8 * time.Second)
        w.Write([]byte("done"))
    })

    srv := &http.Server{
        Addr:         ":8080",
        Handler:      mux,
        ReadTimeout:  15 * time.Second,
        WriteTimeout: 60 * time.Second,
    }

    // Start in a goroutine so we can block on signal below
    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("server error: %v", err)
        }
    }()
    log.Println("server started on :8080")

    // Block until SIGINT or SIGTERM
    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    sig := <-quit
    log.Printf("received signal: %s", sig)

    // Give active connections 30 seconds to finish
    ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer cancel()

    if err := srv.Shutdown(ctx); err != nil {
        log.Fatalf("forced shutdown: %v", err)
    }
    log.Println("server exited cleanly")
}

Two things to note. The buffered channel (make(chan os.Signal, 1)) prevents the signal from being dropped if your goroutine isn't ready to receive. And http.ErrServerClosed is the normal return value from ListenAndServe after a clean shutdown — treating it as fatal will mask that you actually shut down correctly.

Tracking background work spawned by handlers

Server.Shutdown() waits for HTTP connections to close, but it does not wait for goroutines your handlers spawn. If a handler fires async work and returns 202, the shutdown will drain the connection and then your goroutine keeps running — potentially after the database pool is closed.

The fix is a sync.WaitGroup that outlives the HTTP server:

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "os/signal"
    "sync"
    "syscall"
    "time"
)

func main() {
    var wg sync.WaitGroup

    mux := http.NewServeMux()
    mux.HandleFunc("/enqueue", func(w http.ResponseWriter, r *http.Request) {
        // wg.Add MUST happen before the goroutine starts — not inside it.
        // If you call Add inside, there's a race where Shutdown sees zero
        // and returns before the goroutine has incremented the counter.
        wg.Add(1)
        go func() {
            defer wg.Done()
            time.Sleep(5 * time.Second) // replace with real async work
            log.Println("background job completed")
        }()
        w.WriteHeader(http.StatusAccepted)
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}

    go func() {
        if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
            log.Fatalf("server error: %v", err)
        }
    }()

    quit := make(chan os.Signal, 1)
    signal.Notify(quit, syscall.SIGINT, syscall.SIGTERM)
    <-quit

    // Phase 1: stop accepting new requests
    httpCtx, httpCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer httpCancel()
    if err := srv.Shutdown(httpCtx); err != nil {
        log.Printf("http shutdown error: %v", err)
    }

    // Phase 2: wait for in-flight background jobs
    done := make(chan struct{})
    go func() {
        wg.Wait()
        close(done)
    }()

    select {
    case <-done:
        log.Println("all background jobs finished")
    case <-time.After(30 * time.Second):
        log.Println("timeout: some background jobs may have been abandoned")
    }
}

The two-phase approach makes the sequence explicit: first refuse new traffic, then drain existing work. This mirrors how load balancers handle drain — and lets you set different timeouts for each phase.

Production edge cases worth handling

Health check conflicts. If your load balancer polls /health, it needs to see 503 before or right as shutdown begins — otherwise it keeps sending traffic to a server that's draining. A clean approach is to set an atomic flag when shutdown starts and check it in the health handler. Kubernetes's readinessProbe will remove the pod from the service endpoint set once it starts failing.

Multiple servers on different ports. If you expose a public API on :8080 and a metrics endpoint on :9090, shut down the public one first, then metrics. Share a single sync.WaitGroup and a parent context across both.

Timeout alignment. Your Server.Shutdown timeout must be less than Kubernetes's terminationGracePeriodSeconds. If they're equal, the pod gets SIGKILL before your shutdown context can return cleanly. A safe rule: set your app shutdown timeout to terminationGracePeriodSeconds - 5.

Testing the shutdown path. Start your server, issue a slow request (curl localhost:8080/slow &), then send SIGTERM (kill -TERM $(pgrep your-binary)). Verify that:

New connections are immediately refused (connection refused error)
The in-flight request completes normally
The process exits with code 0

This smoke test catches the most common mistake: forgetting to start the server in a goroutine, which blocks main before the signal listener is reached.

For a broader security baseline — including timeouts, header hardening, and connection limits — the security hardening checklists we publish cover the settings most commonly skipped under deadline pressure.

The takeaway

Graceful shutdown in Go is under 50 lines of production-ready code, but the ordering matters. Catch SIGTERM with a buffered channel, call srv.Shutdown() with a context timeout, and track background work with a sync.WaitGroup. Miss the ordering on wg.Add() and you'll get a race that shows up only under load. Miss the timeout alignment with Kubernetes and your pods will get SIGKILLed anyway.

The pattern scales: whether you have one handler or fifty, the signal plumbing stays the same. What changes is which goroutines you register with the WaitGroup and how long you're willing to wait for them.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Implementing SBOM Generation in CI/CD Pipelines Automatically

Ayi NEDJIMI — Wed, 21 Oct 2026 10:01:45 +0000

Every dependency you ship is a liability you haven't fully mapped. The Log4Shell and XZ Utils incidents proved that supply chain attacks are real, scalable, and expensive to remediate without a pre-existing inventory. An SBOM — Software Bill of Materials — is that inventory. The problem: most teams treat SBOM generation as a one-time compliance checkbox rather than a continuously updated artifact built on every commit.

What an SBOM Is (and Which Format to Choose)

An SBOM is a machine-readable list of all components in a software artifact: libraries, OS packages, versions, licenses, and the dependency graph between them. Two formats dominate:

CycloneDX — JSON or XML, designed for security use cases, native support in most vulnerability scanners
SPDX — the Linux Foundation standard, more compliance-oriented (required for some US government contracts post-EO 14028)

For security-first workflows, CycloneDX is the pragmatic choice. For supply chain compliance (NTIA, DoD), SPDX may be required. Both formats can coexist — Syft can emit both from the same scan run.

Generating SBOMs Automatically with Syft

Syft by Anchore is the most widely adopted open-source SBOM generator. It supports Docker images, filesystems, Go binaries, Python wheels, npm trees, and more.

# Install Syft
curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

# Generate SBOM for a Docker image (CycloneDX JSON)
syft your-registry.io/your-app:latest -o cyclonedx-json=sbom.cdx.json

# Generate SBOM for a Go or Python project directory
syft dir:. -o cyclonedx-json=sbom.cdx.json

# Emit both formats simultaneously
syft dir:. \
  -o cyclonedx-json=sbom.cdx.json \
  -o spdx-json=sbom.spdx.json

Syft walks every layer of a container image and every package manifest in a source tree. The output includes PURL (Package URL) identifiers for each component — the standard format that vulnerability databases like OSV.dev and Grype use to match packages against CVE records.

Querying Vulnerabilities from the SBOM

An SBOM without vulnerability correlation is just documentation. The real value is failing the CI build automatically when a critical CVE appears in one of your dependencies.

This Python script reads a CycloneDX JSON SBOM and queries the OSV.dev API for known vulnerabilities against each component's PURL:

import json
import sys
import httpx

CRITICAL_SEVERITIES = {"CRITICAL", "HIGH"}


def query_osv(purl: str) -> list[dict]:
    resp = httpx.post(
        "https://api.osv.dev/v1/query",
        json={"package": {"purl": purl}},
        timeout=10,
    )
    resp.raise_for_status()
    return resp.json().get("vulns", [])


def check_sbom(sbom_path: str, fail_on: set[str] = CRITICAL_SEVERITIES) -> int:
    with open(sbom_path) as f:
        sbom = json.load(f)

    components = sbom.get("components", [])
    findings = []

    for component in components:
        purl = component.get("purl")
        if not purl:
            continue

        for vuln in query_osv(purl):
            severity = vuln.get("database_specific", {}).get("severity", "UNKNOWN").upper()
            if severity in fail_on:
                findings.append({
                    "component": component.get("name"),
                    "version": component.get("version"),
                    "vuln_id": vuln.get("id"),
                    "severity": severity,
                    "summary": vuln.get("summary", ""),
                })

    if findings:
        print(f"\n[FAIL] {len(findings)} critical/high CVEs found:\n")
        for f in findings:
            print(f"  {f['vuln_id']} ({f['severity']}) — {f['component']}@{f['version']}")
            print(f"    {f['summary']}\n")
        return 1

    print(f"[OK] {len(components)} components scanned, no critical/high CVEs.")
    return 0


if __name__ == "__main__":
    path = sys.argv[1] if len(sys.argv) > 1 else "sbom.cdx.json"
    sys.exit(check_sbom(path))

The script exits with code 1 on any HIGH or CRITICAL finding — which fails the CI step. Adjust fail_on to include MEDIUM if your security policy requires it. The OSV.dev API is free, covers PyPI, npm, Go modules, Maven, RubyGems, crates.io, and OS packages from Debian, Alpine, and others.

Wiring It Into GitHub Actions

Here's a complete workflow that builds a Docker image, generates the SBOM, checks vulnerabilities, and uploads the SBOM as a retained artifact:

name: Build and SBOM

on:
  push:
    branches: [main]
  pull_request:

jobs:
  build-and-scan:
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4

      - name: Build image
        run: docker build -t app:${{ github.sha }} .

      - name: Generate SBOM
        uses: anchore/sbom-action@v0
        with:
          image: app:${{ github.sha }}
          format: cyclonedx-json
          output-file: sbom.cdx.json

      - name: Check for critical CVEs
        run: |
          pip install httpx
          python scripts/check_sbom.py sbom.cdx.json

      - name: Store SBOM
        uses: actions/upload-artifact@v4
        with:
          name: sbom-${{ github.sha }}
          path: sbom.cdx.json
          retention-days: 90

A few practical notes on this workflow:

anchore/sbom-action wraps Syft installation and execution in a single step, no manual installation needed
The SBOM is kept for 90 days — useful for post-incident forensics when you need to reconstruct what was running at the time of a breach
Vulnerability checking runs as its own step so the GitHub Actions UI pinpoints exactly where the build failed

Signing and Attesting SBOMs for Audit Trails

Storing an SBOM proves you have one. Attesting it cryptographically proves it was generated from a specific commit by your CI pipeline — not altered after the fact. This matters for compliance frameworks and security hardening audits that require chain-of-custody evidence.

Sigstore's cosign supports keyless attestation via the GitHub Actions OIDC token:

# Attest the SBOM to the image (keyless, uses GitHub OIDC)
cosign attest \
  --type cyclonedx \
  --predicate sbom.cdx.json \
  your-registry.io/your-app:${{ github.sha }}

# Later, verify attestation
cosign verify-attestation \
  --type cyclonedx \
  --certificate-identity-regexp "https://github.com/your-org/.*" \
  --certificate-oidc-issuer "https://token.actions.githubusercontent.com" \
  your-registry.io/your-app:${{ github.sha }}

With keyless signing, cosign requests a short-lived certificate from Fulcio bound to the GitHub Actions job identity, then publishes the attestation to the Rekor transparency log. No key management. Anyone can verify that the SBOM for a given image hash was produced by your CI and not modified.

The Takeaway

SBOM generation is not a one-time deliverable. In a mature DevSecOps pipeline it runs on every build, stores the result, and breaks the build on critical CVEs — fully automatically.

The stack here — Syft, OSV.dev, GitHub Actions, cosign — is entirely open source. The operational cost is near zero beyond CI minutes. Start with generation and the vulnerability gate. Add attestation when you have a consumer for it: a customer contract, a government requirement, or an internal audit.

The harder problem this doesn't solve: what to do when a build breaks. A failed job is only useful if there's a defined process — upgrade the dependency, patch it, or accept the risk with a documented exception. Build the automation first. Then build the process around it.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

How I Cut API Response Times by 60% with Go Caching Strategies

Ayi NEDJIMI — Tue, 20 Oct 2026 10:01:30 +0000

REST APIs that don't cache are burning CPU and database connections on identical work. Once traffic picks up, you feel it fast — latency climbs, the database starts choking, and horizontal scaling just defers the problem. Here's a walkthrough of three caching patterns I applied to a Go service that brought p99 response time from ~420ms down to ~165ms.

The Problem: Too Many Identical Database Round-Trips

Every endpoint was going to the database even when nothing had changed. Cache hit rate was 0%. Profiling was unambiguous: 70% of request time was in SQL queries for rows that hadn't changed in hours. The fix wasn't a faster database — it was not hitting the database at all for identical reads.

Pattern 1: In-Process TTL Cache for Hot Read Paths

Start simple. A generic in-process cache with TTL covers most cases and has zero external dependencies.

package cache

import (
    "sync"
    "time"
)

type entry[V any] struct {
    value     V
    expiresAt time.Time
}

type TTL[K comparable, V any] struct {
    mu    sync.RWMutex
    items map[K]entry[V]
    ttl   time.Duration
}

func New[K comparable, V any](ttl time.Duration) *TTL[K, V] {
    c := &TTL[K, V]{
        items: make(map[K]entry[V]),
        ttl:   ttl,
    }
    go c.runEviction()
    return c
}

func (c *TTL[K, V]) Get(key K) (V, bool) {
    c.mu.RLock()
    defer c.mu.RUnlock()
    e, ok := c.items[key]
    if !ok || time.Now().After(e.expiresAt) {
        var zero V
        return zero, false
    }
    return e.value, true
}

func (c *TTL[K, V]) Set(key K, value V) {
    c.mu.Lock()
    defer c.mu.Unlock()
    c.items[key] = entry[V]{value: value, expiresAt: time.Now().Add(c.ttl)}
}

func (c *TTL[K, V]) runEviction() {
    ticker := time.NewTicker(30 * time.Second)
    for range ticker.C {
        c.mu.Lock()
        now := time.Now()
        for k, e := range c.items {
            if now.After(e.expiresAt) {
                delete(c.items, k)
            }
        }
        c.mu.Unlock()
    }
}

Plug this into any handler layer. For product catalogs, config data, or user preferences where a few seconds of staleness is acceptable, this alone cut database query volume by 45% in my service. The generics make it type-safe without any interface{} juggling.

Pattern 2: singleflight to Prevent Cache Stampedes

In-process TTL caches have a well-known failure mode: when a key expires under burst traffic, dozens of goroutines simultaneously miss the cache and pile into the database. This is a cache stampede, and it can take a service down just when it's under the most pressure.

The standard Go fix is golang.org/x/sync/singleflight:

package handler

import (
    "context"
    "encoding/json"
    "time"

    "golang.org/x/sync/singleflight"

    "yourapp/cache"
    "yourapp/db"
)

var (
    productCache = cache.New[string, []byte](5 * time.Minute)
    sfGroup      singleflight.Group
)

func GetProductsByCategory(ctx context.Context, categoryID string) ([]Product, error) {
    if cached, ok := productCache.Get(categoryID); ok {
        var products []Product
        if err := json.Unmarshal(cached, &products); err == nil {
            return products, nil
        }
    }

    result, err, _ := sfGroup.Do(categoryID, func() (interface{}, error) {
        products, err := db.FetchProductsByCategory(ctx, categoryID)
        if err != nil {
            return nil, err
        }
        if data, marshalErr := json.Marshal(products); marshalErr == nil {
            productCache.Set(categoryID, data)
        }
        return products, nil
    })

    if err != nil {
        return nil, err
    }
    return result.([]Product), nil
}

sfGroup.Do deduplicates concurrent calls for the same key. All waiting goroutines get the result of the single in-flight fetch. Under a traffic spike, this turns hundreds of parallel DB queries into one. The third return value (shared bool) tells you whether the result was actually shared — useful for metrics.

Pattern 3: Tiered Caching with Redis for Multi-Instance Deployments

In-process caches don't survive restarts and don't share state across instances. For data that's expensive to generate and shared across users — aggregations, dashboards, leaderboard rankings — a Redis tier makes sense.

The access pattern is: check in-process cache → check Redis → fetch from DB → write back to both.

package cache

import (
    "context"
    "encoding/json"
    "time"

    "github.com/redis/go-redis/v9"
)

type Tiered[V any] struct {
    local  *TTL[string, V]
    remote *redis.Client
    prefix string
}

func NewTiered[V any](local *TTL[string, V], rdb *redis.Client, prefix string) *Tiered[V] {
    return &Tiered[V]{local: local, remote: rdb, prefix: prefix}
}

func (t *Tiered[V]) Get(ctx context.Context, key string) (V, bool) {
    if v, ok := t.local.Get(key); ok {
        return v, true
    }

    data, err := t.remote.Get(ctx, t.prefix+":"+key).Bytes()
    if err == nil {
        var v V
        if json.Unmarshal(data, &v) == nil {
            t.local.Set(key, v)
            return v, true
        }
    }

    var zero V
    return zero, false
}

func (t *Tiered[V]) Set(ctx context.Context, key string, value V, ttl time.Duration) error {
    t.local.Set(key, value)
    data, err := json.Marshal(value)
    if err != nil {
        return err
    }
    return t.remote.Set(ctx, t.prefix+":"+key, data, ttl).Err()
}

The local layer absorbs the hot tail. Redis handles cold hits and cross-instance consistency. You get sub-millisecond local hits on the most-requested keys without making each instance an island.

What Not to Cache

A few anti-patterns worth naming:

User-specific write-heavy data. Cache invalidation becomes a correctness problem. If a user updates their profile, your cache has to know about it. The logic complexity often outweighs the gain.

Paginated results backed by changing data. You'll serve inconsistent pages — page 1 reflects state at T0, page 2 reflects state at T0+5min.

Anything requiring real-time accuracy. Inventory counts, account balances, seat availability. Cache TTL should express how much staleness is acceptable, not how fast you want things to feel. A 5-minute TTL on data that must be accurate to the second is a bug.

Security Note on Caching

One thing that's easy to overlook: cached responses can expose data to the wrong user if your authorization layer isn't checked on every request. Caching at the handler level can bypass per-user access checks if you're not careful about cache key design. Always include user or tenant identifiers in cache keys for user-scoped data. I keep a security hardening checklist for Go APIs that covers these patterns if you want a reference.

The Takeaway

The 60% improvement came from three layers applied in sequence:

In-process TTL cache on the hottest read endpoints — immediate, zero dependencies, visible impact.
singleflight to collapse burst cache misses — critical under traffic spikes.
Redis tiered cache for shared aggregations — consistent across instances, survives restarts.

None of this required a rewrite. Profile first to confirm where the time actually goes, then apply the simplest pattern that solves it.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

How to Build a Job Queue in Go Without External Dependencies

Ayi NEDJIMI — Mon, 19 Oct 2026 10:01:57 +0000

Most applications need background job processing at some point — sending emails, processing uploads, generating reports. The instinctive answer is to reach for Redis plus a worker library. But for many services, adding that infrastructure is overkill, and Go's concurrency primitives are sufficient on their own.

Why skip the external queue?

External queues (Redis, RabbitMQ, SQS) are the right answer when you need persistence across restarts, distributed workers across multiple machines, or guaranteed delivery with acknowledgment. But if your jobs can tolerate in-process execution and you're running a single instance (or even a few), a channel-based job queue gives you:

Zero additional infrastructure
No network round-trips between enqueue and process
Simple deployment — it's just your binary
Full type safety

The catch: jobs in memory are lost if the process crashes. For many workloads — sending notifications, triggering webhooks, processing user requests — that tradeoff is acceptable.

The core abstraction

A job queue in Go needs three things: a job type, a channel to pass jobs through, and workers consuming from that channel.

package jobqueue

import (
    "context"
    "log/slog"
    "sync"
)

// Job is any function that can be executed asynchronously.
type Job func(ctx context.Context) error

// Queue manages a pool of workers processing jobs from a channel.
type Queue struct {
    jobs    chan Job
    workers int
    wg      sync.WaitGroup
}

// New creates a Queue with the given worker count and buffer size.
func New(workers, bufferSize int) *Queue {
    return &Queue{
        jobs:    make(chan Job, bufferSize),
        workers: workers,
    }
}

// Start launches the worker goroutines.
func (q *Queue) Start(ctx context.Context) {
    for i := 0; i < q.workers; i++ {
        q.wg.Add(1)
        go q.worker(ctx)
    }
}

func (q *Queue) worker(ctx context.Context) {
    defer q.wg.Done()
    for {
        select {
        case job, ok := <-q.jobs:
            if !ok {
                return
            }
            if err := job(ctx); err != nil {
                slog.Error("job failed", "error", err)
            }
        case <-ctx.Done():
            return
        }
    }
}

// Enqueue adds a job to the queue. Returns false if the queue is full.
func (q *Queue) Enqueue(job Job) bool {
    select {
    case q.jobs <- job:
        return true
    default:
        return false
    }
}

// Stop closes the job channel and waits for all workers to finish.
func (q *Queue) Stop() {
    close(q.jobs)
    q.wg.Wait()
}

A few design decisions worth explaining:

Job is func(ctx context.Context) error rather than an interface. This is simpler — you pass closures directly without defining structs for each job type.
Enqueue returns a bool instead of blocking. If the queue is full, the caller decides what to do (log, return HTTP 429, drop) rather than blocking indefinitely.
The ctx.Done() case in the worker handles context cancellation separately from channel close — this matters if you cancel the context before draining the queue.

Wiring it into an HTTP server

Here's a minimal server that uses this queue for background email sending, including a correct graceful shutdown:

package main

import (
    "context"
    "fmt"
    "log"
    "log/slog"
    "net/http"
    "os"
    "os/signal"
    "syscall"
    "time"

    "yourmodule/jobqueue"
)

func main() {
    ctx, cancel := context.WithCancel(context.Background())
    defer cancel()

    q := jobqueue.New(5, 100) // 5 workers, buffer of 100 jobs
    q.Start(ctx)

    mux := http.NewServeMux()
    mux.HandleFunc("/register", func(w http.ResponseWriter, r *http.Request) {
        email := r.FormValue("email")
        if email == "" {
            http.Error(w, "missing email", http.StatusBadRequest)
            return
        }

        enqueued := q.Enqueue(func(ctx context.Context) error {
            slog.Info("sending welcome email", "to", email)
            time.Sleep(50 * time.Millisecond)
            return nil
        })

        if !enqueued {
            http.Error(w, "server busy", http.StatusTooManyRequests)
            return
        }
        fmt.Fprintln(w, "registered")
    })

    srv := &http.Server{Addr: ":8080", Handler: mux}

    go func() {
        if err := srv.ListenAndServe(); err != http.ErrServerClosed {
            log.Fatal(err)
        }
    }()

    sig := make(chan os.Signal, 1)
    signal.Notify(sig, syscall.SIGTERM, syscall.SIGINT)
    <-sig

    shutdownCtx, shutdownCancel := context.WithTimeout(context.Background(), 30*time.Second)
    defer shutdownCancel()

    if err := srv.Shutdown(shutdownCtx); err != nil {
        log.Printf("server shutdown error: %v", err)
    }

    q.Stop() // wait for in-flight jobs
    slog.Info("shutdown complete")
}

The shutdown sequence matters: stop accepting new HTTP requests first, then drain the queue. If you reverse the order, new jobs can arrive after the channel is closed — that panics on a send to a closed channel.

Adding retry logic

Production jobs fail. The minimal version above logs the error and moves on. If you need retries, wrap the job at the call site rather than baking retry logic into the worker:

func withRetry(attempts int, delay time.Duration, job jobqueue.Job) jobqueue.Job {
    return func(ctx context.Context) error {
        var err error
        for i := 0; i < attempts; i++ {
            if err = job(ctx); err == nil {
                return nil
            }
            if i < attempts-1 {
                select {
                case <-time.After(delay * time.Duration(i+1)):
                case <-ctx.Done():
                    return ctx.Err()
                }
            }
        }
        return fmt.Errorf("job failed after %d attempts: %w", attempts, err)
    }
}

// Usage:
q.Enqueue(withRetry(3, 500*time.Millisecond, func(ctx context.Context) error {
    return sendEmail(ctx, "user@example.com")
}))

The time.Duration(i+1) multiplier gives you linear backoff: 500ms, 1s, 1.5s. The ctx.Done() select inside the sleep ensures the job respects cancellation instead of waiting out the full delay on shutdown.

When this design breaks

This approach has real limits. Knowing them in advance saves you from a bad surprise in production.

Process restart loses all queued jobs. If you deploy mid-queue or the process crashes, everything in the channel is gone. If that's unacceptable, you need a persistent queue backed by a database or message broker.

No visibility into the queue. You can add atomic.Int64 counters for enqueued, completed, and failed counts and expose them via a /metrics endpoint, but you won't get job-level introspection without more code.

No priority lanes. A single channel is FIFO. If you need urgent jobs to preempt slow ones, you need multiple queues at different priority levels with a dispatcher that drains high-priority first.

Memory is bounded by your buffer size. Monitor your queue depth in production — a consistently full queue means either your workers are too slow or your traffic exceeds what this pattern can handle, and you need to scale out or switch to a distributed solution.

For teams running job queues inside security tooling or alerting pipelines, make sure your deployment configuration is solid too — we publish free security hardening checklists covering application server and container configurations that are easy to miss.

The takeaway

A channel-based job queue in Go handles a wide range of async workloads with zero operational overhead. The implementation above is around 60 lines, shuts down correctly, and is straightforward to test with a few goroutines and a timeout. Add a retry wrapper when you need resilience. Reach for Redis or a message broker only when you actually need persistence, distributed workers, or job introspection — not as the default starting point.

Start with the simplest thing that could work. In Go, that's often a buffered channel and a handful of goroutines.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.

Go error handling patterns that scale: wrapping, unwrapping, sentinels

Ayi NEDJIMI — Sun, 18 Oct 2026 10:01:32 +0000

Go's error handling gets a lot of criticism from developers coming from exception-based languages. But the real problem isn't if err != nil — it's the patterns that get built on top of it. Most tutorials show you bare error returns and call it a day. In a production codebase, you end up with errors that tell you something failed but not where, why, or what caller context led there.

The Problem with Bare Error Returns

Here's the code you write when you're moving fast:

func getUser(id string) (*User, error) {
    row := db.QueryRow("SELECT * FROM users WHERE id = $1", id)
    var u User
    if err := row.Scan(&u.ID, &u.Name, &u.Email); err != nil {
        return nil, err
    }
    return &u, nil
}

This compiles fine. It even works. But when the error reaches your HTTP handler and you log it, you get:

sql: no rows in result set

That's all. No package name, no function name, no request ID, no context about what was being fetched. Multiply this by 50 functions across 15 packages and you have a log that's useless for debugging.

Wrapping Errors with fmt.Errorf and %w

Go 1.13 introduced the %w verb for fmt.Errorf, which wraps an error while preserving the original for later inspection:

func getUser(id string) (*User, error) {
    row := db.QueryRow("SELECT * FROM users WHERE id = $1", id)
    var u User
    if err := row.Scan(&u.ID, &u.Name, &u.Email); err != nil {
        return nil, fmt.Errorf("getUser(id=%s): %w", id, err)
    }
    return &u, nil
}

Now the error travels up the stack with context attached at each layer:

handler: processRequest: getUser(id=abc123): sql: no rows in result set

The rule I follow: always wrap at the point of origin, never in the middle without adding information. Wrapping fmt.Errorf("getUser: %w", err) inside a loop that calls getUser ten times just adds noise.

One subtle gotcha: %w creates a chain. fmt.Errorf("context: %w", fmt.Errorf("inner: %w", originalErr)) gives you three levels. That's fine — errors.Unwrap navigates the chain, and errors.Is/errors.As traverse it automatically.

Sentinel Errors: When to Use Them

A sentinel error is a package-level variable that callers can compare against:

var ErrNotFound = errors.New("not found")
var ErrUnauthorized = errors.New("unauthorized")

func getUser(id string) (*User, error) {
    row := db.QueryRow("SELECT * FROM users WHERE id = $1", id)
    var u User
    if err := row.Scan(&u.ID, &u.Name, &u.Email); err != nil {
        if errors.Is(err, sql.ErrNoRows) {
            return nil, fmt.Errorf("getUser(id=%s): %w", id, ErrNotFound)
        }
        return nil, fmt.Errorf("getUser(id=%s): %w", id, err)
    }
    return &u, nil
}

The caller can now do:

user, err := getUser(id)
if errors.Is(err, ErrNotFound) {
    http.Error(w, "user not found", http.StatusNotFound)
    return
}

This works because errors.Is traverses the entire error chain — even though ErrNotFound is wrapped inside fmt.Errorf, errors.Is finds it.

When to use sentinels: when callers need to make branching decisions based on the error type. Not found, permission denied, rate limited — these are meaningful categories that affect control flow.

When not to: for every internal error. Package io has io.EOF because it's a legitimate signal, not just a failure. Your internal database timeout doesn't need to be a sentinel unless callers genuinely need to handle it differently.

Custom Error Types for Structured Data

Sometimes you need to carry structured data in an error — an HTTP status code, a field name that failed validation, a request ID for correlation. That's where custom error types come in:

type ValidationError struct {
    Field   string
    Message string
}

func (e *ValidationError) Error() string {
    return fmt.Sprintf("validation failed on field %q: %s", e.Field, e.Message)
}

func validateEmail(email string) error {
    if !strings.Contains(email, "@") {
        return &ValidationError{Field: "email", Message: "must contain @"}
    }
    return nil
}

The caller uses errors.As to extract the structured data:

err := validateEmail(input)
var ve *ValidationError
if errors.As(err, &ve) {
    respondWithFieldError(w, ve.Field, ve.Message)
}

errors.As also traverses the chain, so if validateEmail's error gets wrapped by an outer function, errors.As still finds it.

The combination of custom types and wrapping lets you do something powerful: an HTTP handler can catch a *ValidationError and return 400, catch ErrNotFound and return 404, and return 500 for everything else — without knowing the internals of the packages it calls.

Unwrapping: errors.Is, errors.As, and the Chain

Here's the mental model that makes all of this click:

errors.Is(err, target) — walks the chain calling == at each step. Use for sentinel comparison.
errors.As(err, &target) — walks the chain looking for a type match. Use for custom types.
errors.Unwrap(err) — returns the next error in the chain. Rarely call this directly.

// Chain: wrapErr → ErrNotFound
wrapErr := fmt.Errorf("some context: %w", ErrNotFound)

fmt.Println(errors.Is(wrapErr, ErrNotFound))   // true
fmt.Println(wrapErr == ErrNotFound)             // false — never use == on wrapped errors

This is the most common mistake I see in Go codebases: using == to compare errors after wrapping. It always returns false. Always use errors.Is.

One more thing: if you implement a custom error type that wraps another, implement Unwrap() error to make the chain traversable:

type AppError struct {
    Code    int
    Message string
    Err     error
}

func (e *AppError) Error() string {
    return fmt.Sprintf("[%d] %s: %v", e.Code, e.Message, e.Err)
}

func (e *AppError) Unwrap() error {
    return e.Err
}

Without Unwrap, errors.Is and errors.As stop at AppError and can't see what's wrapped inside it.

Sanitizing Errors at the API Boundary

For security-hardened applications, consistent error handling isn't just about debuggability — it's about not leaking internal details to API consumers while still giving operators enough context to diagnose issues.

A pattern I've used in production:

type internalError struct {
    Op  string
    Err error
}

func (e *internalError) Error() string {
    return fmt.Sprintf("op=%s: %v", e.Op, e.Err)
}

func (e *internalError) Unwrap() error { return e.Err }

// At the HTTP boundary, sanitize before responding
func sanitizeError(err error) string {
    switch {
    case errors.Is(err, ErrNotFound):
        return "resource not found"
    case errors.Is(err, ErrUnauthorized):
        return "unauthorized"
    default:
        log.Printf("internal error: %v", err) // full chain logged internally
        return "internal server error"
    }
}

The internal error carries full context for your logs. The API caller gets a safe, generic message. The operator log has everything needed to trace the failure back to its source.

The Takeaway

Go error handling scales when you follow three rules consistently:

Wrap with context using fmt.Errorf("op: %w", err) at every layer that adds meaningful information.
Use sentinels only for errors that callers genuinely need to branch on — not for every internal failure.
Use custom types when you need structured data alongside the error message, and always implement Unwrap() when your type wraps another error.

The errors.Is and errors.As functions handle chain traversal automatically. Once you internalize that model, Go error handling stops being a boilerplate tax and starts being a precise contract between layers.

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.