Have you ever left an AI agent running in a loop, went to grab a cup of coffee, and came back to a credit card bill that looked like a telephone number?
Yeah. Me too.
AI agents are brilliant, but they have the financial discipline of a golden retriever in a tennis ball factory. They will happily call gpt-4o ten thousand times in a recursive loop to decide if a semicolon is friendly or aggressive.
If you are running LLMs in production, you quickly run into the three horsemen of the AI apocalypse:
- The Infinite Loop Tax: Your code has a bug, the agent panics, and it burns through your API keys at Mach 5.
- Payload Chaos: OpenAI, Anthropic, and Gemini all speak different dialects. Writing schema translation code inside your clean backend microservices feels like eating soup with a fork.
- Connection Cracks: Traditional application-level proxies start weeping when you ask them to handle thousands of concurrent, long-running streaming HTTP connections.
To protect my wallet and sanity, I built a custom LLM Gateway using Go 1.22 and Envoy Proxy. It intercepts traffic at the network edge, translates payloads dynamically, and cuts off greedy clients before they bankrupt me.
Here is how it works, and how you can build one too.
The Blueprint: Envoy + a gRPC Sidecar
Instead of letting backend applications talk directly to OpenAI or Anthropic, we route all outbound requests through a local Envoy Proxy.
Think of Envoy as a highly disciplined traffic controller. It takes the requests, holds the connections, and terminates TLS. But Envoy doesnβt know what a "prompt" is.
To give it brains, we configure Envoy's **External Processing (ext_proc) **filter. This tells Envoy to stream request headers and body chunks over a local gRPC connection to a sidecar service built in Go.
Why this setup is neat:
- Zero Overhead: Envoy handles the massive multiplexing, and our Go sidecar processes the stream in under a millisecond.
- Fail-Closed Security: If the Go sidecar crashes or gets overloaded, Envoy blocks outgoing calls. No sidecar = no spending.
- Microservice Peace: Your backend services talk to a single local endpoint using a unified JSON schema. No more importing six different SDKs.
1. Setting Up Envoy's Filter Rails
We need to tell Envoy: "Hey, whenever you see traffic going to /v1/chat/completions, freeze it, send it to the Go sidecar on port 50051, and wait for approval."
Here is the setup in envoy.yaml:
static_resources:
listeners:
- name: egress_llm_listener
address:
socket_address: { address: 0.0.0.0, port_value: 10000 }
filter_chains:
- filters:
- name: envoy.filters.network.http_connection_manager
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
stat_prefix: ingress_http
route_config:
name: llm_route
virtual_hosts:
- name: llm_service
domains: ["*"]
routes:
- match: { prefix: "/v1/chat/completions" }
route:
cluster: openai_service
timeout: 60s
http_filters:
# Intercept request headers and bodies
- name: envoy.filters.http.ext_proc
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.ext_proc.v3.ExternalProcessor
grpc_service:
envoy_grpc:
cluster_name: go_sidecar_service
failure_mode_allow: false # If sidecar is down, freeze spending
processing_mode:
request_header_mode: "SEND"
response_header_mode: "SEND"
request_body_mode: "BUFFERED"
response_body_mode: "BUFFERED"
- name: envoy.filters.http.router
typed_config:
"@type": type.googleapis.com/envoy.extensions.filters.http.router.v3.Router
clusters:
- name: go_sidecar_service
type: STRICT_DNS
lb_policy: ROUND_ROBIN
http2_protocol_options: {}
load_assignment:
cluster_name: go_sidecar_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: 127.0.0.1, port_value: 50051 }
- name: openai_service
type: LOGICAL_DNS
dns_lookup_family: V4_ONLY
lb_policy: ROUND_ROBIN
transport_socket:
name: envoy.transport_sockets.tls
load_assignment:
cluster_name: openai_service
endpoints:
- lb_endpoints:
- endpoint:
address:
socket_address: { address: api.openai.com, port_value: 443 }
2. Writing the Go gRPC Sidecar
Our Go sidecar implements Envoy's proto contract. We use Go 1.22's native concurrency patterns to process incoming message streams without breaking a sweat.
Here is the server setup:
package main
import (
"context"
"encoding/json"
"fmt"
"io"
"log"
"net"
extproc "github.com/envoyproxy/go-control-plane/envoy/service/ext_proc/v3"
pb "github.com/envoyproxy/go-control-plane/envoy/type/v3"
"google.golang.org/grpc"
)
type server struct {
extproc.UnimplementedExternalProcessorServer
}
func (s *server) Process(stream extproc.ExternalProcessor_ProcessServer) error {
for {
req, err := stream.Recv()
if err == io.EOF {
return nil
}
if err != nil {
return err
}
var resp *extproc.ProcessingResponse
switch request := req.Request.(type) {
case *extproc.ProcessingRequest_RequestHeaders:
resp = handleRequestHeaders(request.RequestHeaders)
case *extproc.ProcessingRequest_RequestBody:
resp = handleRequestBody(request.RequestBody)
default:
// Passthrough for things we don't care about
resp = &extproc.ProcessingResponse{
Response: &extproc.ProcessingResponse_CommonResponse{
CommonResponse: &extproc.CommonResponse{},
},
}
}
if err := stream.Send(resp); err != nil {
return err
}
}
}
func main() {
lis, err := net.Listen("tcp", ":50051")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
s := grpc.NewServer()
extproc.RegisterExternalProcessorServer(s, &server{})
log.Println("gRPC External Processor is alive on :50051...")
if err := s.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
3. Translation on the Fly
If your developer wants to target Claude but sent an OpenAI-shaped JSON payload, the sidecar intercepts the request buffer, rips it open, maps it to Anthropic's structure, and updates Envoy's headers.
Here is the mapping code:
type UnifiedRequest struct {
Model string `json:"model"`
Prompt string `json:"prompt"`
MaxTokens int `json:"max_tokens"`
}
type AnthropicRequest struct {
Model string `json:"model"`
Messages []Message `json:"messages"`
MaxTokens int `json:"max_tokens"`
}
type Message struct {
Role string `json:"role"`
Content string `json:"content"`
}
func handleRequestBody(body *extproc.HttpBody) *extproc.ProcessingResponse {
var unified UnifiedRequest
if err := json.Unmarshal(body.Body, &unified); err != nil {
return rejectRequest("JSON parsed poorly", 400)
}
// Dynamic translation: Translate OpenAI-like payload to Anthropic format
if unified.Model == "claude-3-opus" {
anthropicPayload := AnthropicRequest{
Model: "claude-3-opus-20240229",
MaxTokens: unified.MaxTokens,
Messages: []Message{
{Role: "user", Content: unified.Prompt},
},
}
newBody, _ := json.Marshal(anthropicPayload)
return &extproc.ProcessingResponse{
Response: &extproc.ProcessingResponse_RequestBody{
RequestBody: &extproc.BodyResponse{
Response: &extproc.CommonResponse{
HeaderMutation: &extproc.HeaderMutation{
SetHeaders: []*pb.HeaderValueOption{
{
Header: &pb.HeaderValue{
Key: "content-length",
Value: fmt.Sprintf("%d", len(newBody)),
},
},
},
},
BodyMutation: &extproc.BodyMutation{
Mutation: &extproc.BodyMutation_Body{
Body: newBody,
},
},
},
},
},
}
}
// Otherwise, let it go through untouched
return &extproc.ProcessingResponse{
Response: &extproc.ProcessingResponse_RequestBody{
RequestBody: &extproc.BodyResponse{},
},
}
}
func rejectRequest(reason string, statusCode int32) *extproc.ProcessingResponse {
return &extproc.ProcessingResponse{
Response: &extproc.ProcessingResponse_ImmediateResponse{
ImmediateResponse: &extproc.ImmediateResponse{
Status: &pb.HttpStatus{Code: pb.StatusCode(statusCode)},
Body: reason,
},
},
}
}
4. Setting the Financial Guardrails
To stop runaway budget loops, we check client spend in handleRequestHeaders. We look for the X-Client-ID header and verify their daily balance against Redis:
var ctx = context.Background()
func handleRequestHeaders(headers *extproc.HttpHeaders) *extproc.ProcessingResponse {
var clientID string
for _, h := range headers.Headers.Headers {
if h.Key == "x-client-id" {
clientID = h.Value
break
}
}
if clientID == "" {
return rejectRequest("Who are you? (Missing X-Client-ID header)", 401)
}
// Query Redis
spent, err := redisClient.Get(ctx, fmt.Sprintf("budget:%s", clientID)).Float64()
if err != nil && err != redis.Nil {
return rejectRequest("Rate limit engine hiccuped", 500)
}
limit := 50.00 // $50 daily limit
if spent >= limit {
return rejectRequest("Your agent spent too much money today. No more queries.", 429)
}
return &extproc.ProcessingResponse{
Response: &extproc.ProcessingResponse_RequestHeaders{
RequestHeaders: &extproc.HeadersResponse{},
},
}
}
When the response streams back from the LLM, the sidecar catches it, reads the token usage metadata, updates the Redis balance in a background goroutine, and forwards it to the client.
Speed Benchmarks
Adding a proxy step sounds like it would add latency. But because we are using Go's lightweight runtime threads (goroutines) and Envoy's raw HTTP/2 connection pooling, the overhead is tiny:
- Direct call to LLM: ~1250ms (inference takes 99.9% of the time)
- Proxy call (Envoy + Go Sidecar): ~1250.7ms
- Total Gateway Overhead: <0.8ms
Wrapping Up
Building a network-edge gateway gives you absolute control over your API spend and mapping, protecting you from schema updates and loops.
Have you built a gateway or system to keep your LLM costs from blowing up? Letβs swap war stories in the comments below.
For more systems stuff and code packages, visit my portfolio or check my GitHub @nyx-abu.
Top comments (0)