Why I Built an LLM Gateway (Because My AI Agents Have No Concept of Money)

Abdur raheem — Tue, 07 Jul 2026 22:55:15 +0000

Have you ever left an AI agent running in a loop, went to grab a cup of coffee, and came back to a credit card bill that looked like a telephone number?

Yeah. Me too.

AI agents are brilliant, but they have the financial discipline of a golden retriever in a tennis ball factory. They will happily call gpt-4o ten thousand times in a recursive loop to decide if a semicolon is friendly or aggressive.

If you are running LLMs in production, you quickly run into the three horsemen of the AI apocalypse:

The Infinite Loop Tax: Your code has a bug, the agent panics, and it burns through your API keys at Mach 5.
Payload Chaos: OpenAI, Anthropic, and Gemini all speak different dialects. Writing schema translation code inside your clean backend microservices feels like eating soup with a fork.
Connection Cracks: Traditional application-level proxies start weeping when you ask them to handle thousands of concurrent, long-running streaming HTTP connections.

To protect my wallet and sanity, I built a custom LLM Gateway using Go 1.22 and Envoy Proxy. It intercepts traffic at the network edge, translates payloads dynamically, and cuts off greedy clients before they bankrupt me.

Here is how it works, and how you can build one too.

The Blueprint: Envoy + a gRPC Sidecar

Instead of letting backend applications talk directly to OpenAI or Anthropic, we route all outbound requests through a local Envoy Proxy.

Think of Envoy as a highly disciplined traffic controller. It takes the requests, holds the connections, and terminates TLS. But Envoy doesn’t know what a "prompt" is.

To give it brains, we configure Envoy's **External Processing (ext_proc) **filter. This tells Envoy to stream request headers and body chunks over a local gRPC connection to a sidecar service built in Go.

Why this setup is neat:

Zero Overhead: Envoy handles the massive multiplexing, and our Go sidecar processes the stream in under a millisecond.
Fail-Closed Security: If the Go sidecar crashes or gets overloaded, Envoy blocks outgoing calls. No sidecar = no spending.
Microservice Peace: Your backend services talk to a single local endpoint using a unified JSON schema. No more importing six different SDKs.

1. Setting Up Envoy's Filter Rails

We need to tell Envoy: "Hey, whenever you see traffic going to /v1/chat/completions, freeze it, send it to the Go sidecar on port 50051, and wait for approval."

Here is the setup in envoy.yaml:

static_resources:
  listeners:
  - name: egress_llm_listener
    address:
      socket_address: { address: 0.0.0.0, port_value: 10000 }
    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            name: llm_route
            virtual_hosts:
            - name: llm_service
              domains: ["*"]
              routes:
              - match: { prefix: "/v1/chat/completions" }
                route:
                  cluster: openai_service
                  timeout: 60s
          http_filters:
          # Intercept request headers and bodies
          - name: envoy.filters.http.ext_proc
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
              grpc_service:
                envoy_grpc:
                  cluster_name: go_sidecar_service
              failure_mode_allow: false # If sidecar is down, freeze spending
              processing_mode:
                request_header_mode: "SEND"
                response_header_mode: "SEND"
                request_body_mode: "BUFFERED"
                response_body_mode: "BUFFERED"
          - name: envoy.filters.http.router
            typed_config:
              "@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router

  clusters:
  - name: go_sidecar_service
    type: STRICT_DNS
    lb_policy: ROUND_ROBIN
    http2_protocol_options: {}
    load_assignment:
      cluster_name: go_sidecar_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: 127.0.0.1, port_value: 50051 }

  - name: openai_service
    type: LOGICAL_DNS
    dns_lookup_family: V4_ONLY
    lb_policy: ROUND_ROBIN
    transport_socket:
      name: envoy.transport_sockets.tls
    load_assignment:
      cluster_name: openai_service
      endpoints:
      - lb_endpoints:
        - endpoint:
            address:
              socket_address: { address: api.openai.com, port_value: 443 }

2. Writing the Go gRPC Sidecar

Our Go sidecar implements Envoy's proto contract. We use Go 1.22's native concurrency patterns to process incoming message streams without breaking a sweat.

Here is the server setup:

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net"

    extproc "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
    pb "github.com/envoyproxy/go-control-plane/envoy/type/v3"
    "google.golang.org/grpc"
)

type server struct {
    extproc.UnimplementedExternalProcessorServer
}

func (s *server) Process(stream extproc.ExternalProcessor_ProcessServer) error {
    for {
        req, err := stream.Recv()
        if err == io.EOF {
            return nil
        }
        if err != nil {
            return err
        }

        var resp *extproc.ProcessingResponse

        switch request := req.Request.(type) {
        case *extproc.ProcessingRequest_RequestHeaders:
            resp = handleRequestHeaders(request.RequestHeaders)
        case *extproc.ProcessingRequest_RequestBody:
            resp = handleRequestBody(request.RequestBody)
        default:
            // Passthrough for things we don't care about
            resp = &extproc.ProcessingResponse{
                Response: &extproc.ProcessingResponse_CommonResponse{
                    CommonResponse: &extproc.CommonResponse{},
                },
            }
        }

        if err := stream.Send(resp); err != nil {
            return err
        }
    }
}

func main() {
    lis, err := net.Listen("tcp", ":50051")
    if err != nil {
        log.Fatalf("failed to listen: %v", err)
    }
    s := grpc.NewServer()
    extproc.RegisterExternalProcessorServer(s, &server{})
    log.Println("gRPC External Processor is alive on :50051...")
    if err := s.Serve(lis); err != nil {
        log.Fatalf("failed to serve: %v", err)
    }
}

3. Translation on the Fly

If your developer wants to target Claude but sent an OpenAI-shaped JSON payload, the sidecar intercepts the request buffer, rips it open, maps it to Anthropic's structure, and updates Envoy's headers.

Here is the mapping code:

type UnifiedRequest struct {
    Model       string `json:"model"`
    Prompt      string `json:"prompt"`
    MaxTokens   int    `json:"max_tokens"`
}

type AnthropicRequest struct {
    Model     string    `json:"model"`
    Messages  []Message `json:"messages"`
    MaxTokens int       `json:"max_tokens"`
}

type Message struct {
    Role    string `json:"role"`
    Content string `json:"content"`
}

func handleRequestBody(body *extproc.HttpBody) *extproc.ProcessingResponse {
    var unified UnifiedRequest
    if err := json.Unmarshal(body.Body, &unified); err != nil {
        return rejectRequest("JSON parsed poorly", 400)
    }

    // Dynamic translation: Translate OpenAI-like payload to Anthropic format
    if unified.Model == "claude-3-opus" {
        anthropicPayload := AnthropicRequest{
            Model:     "claude-3-opus-20240229",
            MaxTokens: unified.MaxTokens,
            Messages: []Message{
                {Role: "user", Content: unified.Prompt},
            },
        }

        newBody, _ := json.Marshal(anthropicPayload)
        return &extproc.ProcessingResponse{
            Response: &extproc.ProcessingResponse_RequestBody{
                RequestBody: &extproc.BodyResponse{
                    Response: &extproc.CommonResponse{
                        HeaderMutation: &extproc.HeaderMutation{
                            SetHeaders: []*pb.HeaderValueOption{
                                {
                                    Header: &pb.HeaderValue{
                                        Key:   "content-length",
                                        Value: fmt.Sprintf("%d", len(newBody)),
                                    },
                                },
                            },
                        },
                        BodyMutation: &extproc.BodyMutation{
                            Mutation: &extproc.BodyMutation_Body{
                                Body: newBody,
                            },
                        },
                    },
                },
            },
        }
    }

    // Otherwise, let it go through untouched
    return &extproc.ProcessingResponse{
        Response: &extproc.ProcessingResponse_RequestBody{
            RequestBody: &extproc.BodyResponse{},
        },
    }
}

func rejectRequest(reason string, statusCode int32) *extproc.ProcessingResponse {
    return &extproc.ProcessingResponse{
        Response: &extproc.ProcessingResponse_ImmediateResponse{
            ImmediateResponse: &extproc.ImmediateResponse{
                Status: &pb.HttpStatus{Code: pb.StatusCode(statusCode)},
                Body:   reason,
            },
        },
    }
}

4. Setting the Financial Guardrails

To stop runaway budget loops, we check client spend in handleRequestHeaders. We look for the X-Client-ID header and verify their daily balance against Redis:

var ctx = context.Background()

func handleRequestHeaders(headers *extproc.HttpHeaders) *extproc.ProcessingResponse {
    var clientID string
    for _, h := range headers.Headers.Headers {
        if h.Key == "x-client-id" {
            clientID = h.Value
            break
        }
    }

    if clientID == "" {
        return rejectRequest("Who are you? (Missing X-Client-ID header)", 401)
    }

    // Query Redis
    spent, err := redisClient.Get(ctx, fmt.Sprintf("budget:%s", clientID)).Float64()
    if err != nil && err != redis.Nil {
        return rejectRequest("Rate limit engine hiccuped", 500)
    }

    limit := 50.00 // $50 daily limit
    if spent >= limit {
        return rejectRequest("Your agent spent too much money today. No more queries.", 429)
    }

    return &extproc.ProcessingResponse{
        Response: &extproc.ProcessingResponse_RequestHeaders{
            RequestHeaders: &extproc.HeadersResponse{},
        },
    }
}

When the response streams back from the LLM, the sidecar catches it, reads the token usage metadata, updates the Redis balance in a background goroutine, and forwards it to the client.

Speed Benchmarks

Adding a proxy step sounds like it would add latency. But because we are using Go's lightweight runtime threads (goroutines) and Envoy's raw HTTP/2 connection pooling, the overhead is tiny:

Direct call to LLM: ~1250ms (inference takes 99.9% of the time)
Proxy call (Envoy + Go Sidecar): ~1250.7ms
Total Gateway Overhead: <0.8ms

Wrapping Up

Building a network-edge gateway gives you absolute control over your API spend and mapping, protecting you from schema updates and loops.

Have you built a gateway or system to keep your LLM costs from blowing up? Let’s swap war stories in the comments below.

For more systems stuff and code packages, visit my portfolio or check my GitHub @nyx-abu.

DEV Community: Abdur raheem