DEV Community: Stoney Epling

Why We Built an AI Gateway in Go: Failover, PII Redaction, and Sub-Millisecond Caching

Stoney Epling — Wed, 02 Sep 2026 17:06:31 +0000

Every team shipping LLMs to production hits the same three bottlenecks sooner or later:

Upstream Downtime & Spikes: An OpenAI 503 or an Anthropic 529 "Overloaded" error takes down your user-facing app.
Compliance & Data Leaks: Sensitive customer data (emails, credit cards, SSNs) gets sent to external foundation model providers unredacted.
Runaway Token Costs: Repetitive, near-identical prompts run through high-cost frontier models instead of hitting an in-memory cache.

To solve this, many teams turn to proxies like LiteLLM, Portkey, or Helicone. While these tools have paved the way, we found recurring operational friction around multi-tenant quota desyncs, high proxy overhead, and complex self-hosting setups.

We built SentinelGateway as a zero-dependency, high-throughput Go proxy that acts as an OpenAI-compatible drop-in layer for production apps.

Here is an architectural breakdown of how it works under the hood.

1. Zero Codebase Refactoring (The 1-Line Drop-In)

Developers should not have to learn a proprietary SDK or rewrite orchestration chains. SentinelGateway implements the complete /v1/chat/completions schema.

Switching from direct OpenAI calls to multi-provider proxying requires updating only the base configuration:

import os
from openai import OpenAI

# Drop-in SentinelGateway proxy
client = OpenAI(
    base_url="[https://sentinelgateway.ai/v1](https://sentinelgateway.ai/v1)",
    api_key=os.environ.get("SENTINEL_GATEWAY_KEY")
)

response = client.chat.completions.create(
    model="gpt-4-failover",
    messages=[{"role": "user", "content": "Process this document..."}]
)

2. Multi-Provider Capacity-Aware Failover

When routing across OpenAI, Anthropic, Groq, and Google Gemini, raw network retries are not enough. If OpenAI returns HTTP 429 (rate-limited) or HTTP 503 (service unavailable), a naive retry to the same endpoint only amplifies cascading failures.

SentinelGateway isolates upstream adapters and runs a capacity-aware failover loop:

The gateway checks upstream health and rate allocations in Redis.
If the primary provider fails or returns a recoverable 5xx or 429 status, the proxy catches the error before the client socket drops.
The prompt is automatically translated into the target provider's native format (e.g., Anthropic Messages API or Gemini REST payload) and dispatched in sub-25ms.
If upstream token consumption fails midway, reservation refunds execute atomically to prevent phantom billing.

3. In-Flight, Zero-Retention PII Scrubbing

Sending raw prompts containing user identifiers to upstream providers creates serious compliance liabilities under GDPR and SOC2.

SentinelGateway intercepts payloads in-flight before they touch the wire:

High-speed token scanners detect SSNs, credit card numbers, API keys, and email addresses.
Sensitive entities are redacted or pseudonymized in memory.
Zero retention: The gateway does not write prompt bodies to permanent disk logs, eliminating downstream breach exposure.

4. Sub-Millisecond Semantic Caching in Redis

Many production workloads—such as customer support bots, classification pipelines, and documentation copilots—process semantically identical queries repeatedly.

SentinelGateway runs an atomic Redis Lua evaluation pipeline:

Exact and normalized prompt fingerprints are verified against hot cache tiers.
Cache hits return in sub-15ms with zero upstream token consumption.
Monthly quotas are decremented using atomic Lua counters, ensuring strict concurrency isolation across distributed nodes without race conditions.

How Does It Compare?

If you are evaluating AI gateways for your infrastructure, check out our detailed side-by-side architecture comparisons:

Sentinel vs. LiteLLM: A look at Python vs. Go runtimes, throughput benchmarks, and deployment overhead.
Sentinel vs. Portkey: Comparing telemetry pipelines, rate limiting, and enterprise control planes.
Sentinel vs. Helicone: Logging overhead, semantic caching efficiency, and proxy latencies.

Getting Started

SentinelGateway is live in production. You can spin up a free workspace with an included monthly token quota to test failovers and caching in your staging pipeline:

👉 Get Started Free on SentinelGateway

If you are running production AI workloads and have questions about our Go routing architecture or Redis Lua quota scripts, drop a comment below!

Tutorial: How to Detect VPNs and Tor Users in Node.js Express

Stoney Epling — Tue, 16 Dec 2025 03:46:21 +0000

If you run any kind of public API, SaaS, or forum, you already know the pain: Bot traffic.

You ban a user for spamming, and 5 seconds later they are back with a new account because they toggled their VPN. You block an IP, and they switch to a Tor exit node.

In this tutorial, I'm going to show you how to detect Non-Residential IPs (VPNs, Proxies, and Hosting Centers) in your Node.js application so you can block them—or at least challenge them with a CAPTCHA—before they touch your database.

The Goal

We want a middleware function in Express that looks like this:


javascript
app.use((req, res, next) => {
  if (isHighRisk(req.ip)) {
     return res.status(403).send("VPNs are not allowed.");
  }
  next();
});

Here is how to build it.

Method 1: The "Hard" Way (Self-Hosted Lists)
If you want to do this entirely for free and offline, you need to download and maintain lists of known "Bad IPs."

Step 1: Get the Data
You will need to find a text file of Tor Exit nodes and IP ranges for major cloud providers (AWS, DigitalOcean, Linode).

Tor Exit Nodes: The Tor project publishes a list of exit addresses.

Cloud Ranges: AWS and Google publish their IP ranges in massive JSON files.

Step 2: The Code
You'll need to parse these lists into memory and check every incoming request.

JavaScript

const fs = require('fs');
const ipRangeCheck = require('ip-range-check'); // You'll need this npm package

// 1. Load the massive lists into memory (Careful with RAM!)
const torNodes = fs.readFileSync('tor-exit-nodes.txt', 'utf8').split('\n');
const awsRanges = JSON.parse(fs.readFileSync('aws-ip-ranges.json', 'utf8')).prefixes.map(p => p.ip_prefix);

function isHighRisk(userIp) {
    // Check if IP is in the Tor list
    if (torNodes.includes(userIp)) return true;

    // Check if IP is in a Cloud Range (CPU intensive)
    if (ipRangeCheck(userIp, awsRanges)) return true;

    return false;
}

The Problem with Method 1
Stale Data: VPN providers rotate IPs daily. If you don't update your lists every hour, you will miss attacks.

Memory Hog: Loading millions of IPs into Node.js memory can crash your server (I learned this the hard way and OOM-killed my $5 droplet).

False Positives: It's hard to distinguish between a "Good" data center IP and a "Bad" VPN.

Method 2: The "Easy" Way (Live API Lookup)
After struggling with maintaining my own lists, I built a dedicated API called CandyCornDB to handle the heavy lifting. It specifically targets Infrastructure (ASN/ISP data) rather than just "bad behavior," so it catches fresh VPNs instantly.

Here is how to implement it in 3 lines of code.

Step 1: Get a Free API Key
You can grab a free key here (no credit card required).

Step 2: The Middleware
We will query the API, which returns a trustScore (0-100).

0-50: Residential / Safe

75+: High Risk (VPN / Tor / Hosting)

JavaScript

const axios = require('axios');

async function checkRiskScore(req, res, next) {
    const userIp = req.ip; 

    try {
        const response = await axios.get('[https://candycorndb.com/api/public/ip-score](https://candycorndb.com/api/public/ip-score)', {
            params: { ip: userIp }
        });

        const { score, isTor, isVPN } = response.data;

        // BLOCK if it's a confirmed Tor node or very high risk
        if (isTor || score > 85) {
            return res.status(403).json({ error: 'Anonymizers not allowed.' });
        }

        // CHALLENGE if it's suspicious (e.g., DigitalOcean droplet)
        if (score >= 50) {
            // Logic to show a CAPTCHA goes here...
            console.log(`Suspicious traffic from ${userIp}`);
        }

        next();
    } catch (err) {
        // Fail open: If API is down, let the user in so you don't block real people
        next();
    }
}

// Apply to your sensitive routes
app.post('/api/signup', checkRiskScore, (req, res) => {
    res.send("Account created!");
});

Why this is better
Just-in-Time Scanning: If the API hasn't seen the IP before, it scans open ports and ISP data in <500ms. You never get "Unknown."

No Maintenance: You don't need to download daily CSV dumps.

Saves RAM: Your Node server handles the logic, not the database storage.

Summary

Blocking bad IPs is an arms race. If you are building a small hobby project, Method 1 is a fun learning exercise. But if you are protecting a production app, offloading the risk detection to a dedicated API (Method 2) is usually cheaper than the time you'll spend unbanning spam accounts.

Let me know if you have questions about IP filtering logic! I've spent way too much time staring at ASN lists lately. 😅