How I Built an OTel-Native GPU Profiler & LLM Gateway to Stop Bleeding Inference Costs

#n8n #python #ai #architecture

Running LLM workloads at scale—especially when integrating custom AI agents into n8n automation pipelines—creates a massive infrastructure blind spot. You build a brilliant workflow, fire thousands of requests, and suddenly your cloud bill skyrockets because of opaque GPU utilization and runaway inference costs.

Traditional cloud monitoring lacks granular, step-by-step visibility into CUDA kernel execution and per-token generation costs. As a result, engineering teams over-provision expensive hardware.

In this guide, I'll show you how to build an OTel-Native GPU Profiler & LLM Cost Optimization Gateway. We will use this async middleware to sit between an n8n lead enrichment pipeline and your custom AI backend, tracking real-time CUDA memory, calculating per-token inference costs, and exporting metrics natively to Prometheus and Grafana.

1. The Bottleneck: Why commercial solutions are overpriced or fragile

When scaling autonomous workflows (like scraping domains and analyzing them with an LLM to route to a CRM), you quickly hit bottlenecks. Commercial LLM observability tools (like Datadog or New Relic) charge a premium for AI APMs, and they often fail to capture deep hardware-level metrics like live VRAM allocation or step-latency inside PyTorch/vLLM.

If you don't know exactly how many tokens a specific prompt consumes, or how much CUDA memory spikes during a batched request, you end up renting an A100 when an L4 would have sufficed. The solution is an OpenTelemetry (OTel) native metric export running as a drop-in async middleware gateway.

2. The Architecture: Step-by-step breakdown of how the pipeline works

To bridge the gap between n8n's automation logic and raw GPU compute, the architecture is split into three layers:

The Orchestrator (n8n): Pulls raw lead data, triggers the HTTP request to our Gateway, and handles the final CRM routing.
The Async Gateway (FastAPI + OTel): Receives the payload, sanitizes the domain, structures the LLM prompt, and records real-time hardware telemetry.
The Observability Stack: The gateway exports OpenTelemetry metrics directly to a Prometheus scraper, visualized via Grafana.

Instead of n8n talking directly to OpenAI or a bare-metal HuggingFace endpoint, it talks to our Gateway. The Gateway wraps the inference call in an OTel trace, profiles the CUDA GPU, calculates the exact dollar cost of the run, and returns both the inference result and the cost metadata.

3. The Code & Logic: Domain sanitization, LLM prompt structuring, and OTel Metrics

Let's look at the core of the drop-in async middleware gateway. We need to handle the incoming request, sanitize the input data, structure the prompt, and most importantly, profile the GPU.

import time
import torch
from fastapi import FastAPI, Request
from opentelemetry import trace, metrics
from pydantic import BaseModel
from urllib.parse import urlparse

app = FastAPI()

# Initialize OTel Tracer and Meter
tracer = trace.get_tracer(__name__)
meter = metrics.get_meter(__name__)
cost_counter = meter.create_counter("llm.inference.cost", description="Calculated inference cost in USD")
vram_gauge = meter.create_gauge("gpu.cuda.vram_allocated", description="Current VRAM allocated in bytes")

class LeadPayload(BaseModel):
    raw_url: str
    company_name: str

def sanitize_domain(raw_url: str) -> str:
    """Strips UTMs, paths, and protocols to extract the clean root domain."""
    parsed = urlparse(raw_url if "//" in raw_url else f"http://{raw_url}")
    domain = parsed.netloc.replace("www.", "")
    return domain

def build_prompt(domain: str, company: str) -> str:
    """Structures a strict system prompt to prevent LLM hallucinations."""
    return f"""
    System: You are an expert B2B data extraction agent.
    Task: Analyze the company '{company}' at domain '{domain}'.
    Output strict JSON with fields: 'industry', 'employee_estimate', 'tech_stack'.
    """

@app.post("/api/v1/enrich-lead")
async def enrich_lead(payload: LeadPayload):
    with tracer.start_as_current_span("enrich_lead_pipeline") as span:
        # 1. Domain Sanitization
        clean_domain = sanitize_domain(payload.raw_url)
        prompt = build_prompt(clean_domain, payload.company_name)

        # 2. Pre-Inference GPU Profiling
        if torch.cuda.is_available():
            vram_before = torch.cuda.memory_allocated()
            vram_gauge.set(vram_before)
            span.set_attribute("gpu.vram.before", vram_before)

        # 3. Step-latency profiling for LLM execution
        start_time = time.time()

        # Mocking the async LLM call (Replace with vLLM/Transformers logic)
        response_data, tokens_used = await run_local_inference(prompt)

        latency = time.time() - start_time
        span.set_attribute("llm.latency_seconds", latency)

        # 4. Per-token inference cost calculation ($0.0002 per 1k tokens)
        calculated_cost = (tokens_used / 1000) * 0.0002
        cost_counter.add(calculated_cost)
        span.set_attribute("llm.cost_usd", calculated_cost)

        # 5. Post-Inference VRAM check
        if torch.cuda.is_available():
            vram_after = torch.cuda.memory_allocated()
            vram_gauge.set(vram_after)
            span.set_attribute("gpu.vram.after", vram_after)

        return {
            "enriched_data": response_data,
            "profiling": {
                "cost_usd": calculated_cost,
                "latency": latency,
                "tokens": tokens_used
            }
        }

By injecting this logic into the middleware, every single lead enrichment run outputs granular OTel data natively formatted for your Prometheus and Grafana dashboards.

4. Deployment & Performance: Timeouts, Rate Limits, and CRM Routing

Running a local LLM gateway means you are now responsible for the hardware lifecycle. Here is how to handle the critical deployment factors:

Timeout Management: In n8n, set the HTTP Request node timeout to at least 45 seconds to account for cold-start latency or massive prompt token generation. In your Python Gateway, utilize asyncio.wait_for to ensure requests don't hang indefinitely and lock up your VRAM.
Rate Limiting: GPUs queue up fast. Use a tool like Redis with FastAPI's SlowAPI to rate-limit incoming requests from n8n. If n8n attempts to push 500 leads simultaneously, the Gateway should return a 429 Too Many Requests. Configure n8n's "Retry on Fail" settings to use an exponential backoff.
CRM Routing: Once the Gateway responds with the enriched_data, n8n receives the sanitized JSON. You can then use a Switch node in n8n: if the industry is "Software" and the cost was under $0.01, route it immediately to the Salesforce/HubSpot node. If not, append it to a Google Sheet for manual review.

5. Conclusion & Ready-to-use Workflow

You no longer have an excuse for opaque LLM costs. By implementing an OTel-native Gateway, you get real-time CUDA memory tracking, compute utilization metrics, and per-token cost calculations without locking into expensive enterprise APMs.

You can take the code blocks above, integrate them into your FastApi/n8n stack, and build this entire pipeline yourself.

However, if you want to skip the trial-and-error of configuring Prometheus endpoints, writing the Grafana dashboard JSON, and setting up the async chunking logic, I've packaged the entire production-ready system.

Get the complete turnkey package, including test fixtures, n8n workflow JSON, and the full Gateway source code:

Instant Access on Whop: OTel-Native GPU Profiler & LLM Cost Optimization Gateway
Direct Download on Gumroad: Grab the Zip File here — use promo code EARLYBIRD for 20% off.

Stop over-provisioning your GPUs and start optimizing your autonomous pipelines today.

DEV Community