DEV Community: cauqjbwkerl

Why your Claude and OpenAI API calls are slow (and how to fix it)

cauqjbwkerl — Tue, 30 Jun 2026 08:35:49 +0000

TL;DR

If you're calling the Claude or OpenAI API from Asia, Oceania, or South America, you're likely adding 400–800ms of pure network overhead before a single token arrives. The root cause is geography, not the AI model itself — and it's fixable with smarter routing.

The Problem Nobody Talks About

I spent two weeks debugging what I thought was a slow model. My streaming chat app felt sluggish — users were staring at a blank screen for nearly a second before anything appeared. After profiling every layer of my stack, I finally isolated the culprit: the raw time from my server in Tokyo to api.anthropic.com and back was eating 600–900ms before the API even started generating.

This isn't a Claude or OpenAI problem. It's physics and routing.

Why AI API Latency Is Geography-Dependent

Both Anthropic and OpenAI run their inference infrastructure primarily in US data centers (us-east-1, us-west-2 regions). When you're in Singapore, Seoul, or Sydney, every API call has to:

Cross transoceanic fiber — a round-trip from Tokyo to Virginia is ~160ms at the speed of light, and real-world routing adds 30–50% on top of that.
Negotiate TLS from a distance — a TLS 1.3 handshake requires 1 round-trip; TLS 1.2 requires 2. At 200ms RTT, that's 200–400ms gone before a byte of your prompt is sent.
Fight TCP congestion control — long-haul routes traverse multiple ISP handoffs. TCP's slow-start and congestion windows are tuned for short distances; on transoceanic routes, you get retransmits and window stalls that inflate latency unpredictably.

The result: developers in the US see 40–80ms to first token on streaming calls. Developers in Asia routinely see 300–600ms, sometimes spiking past 1 second during peak hours.

Measure Your Baseline First

Before optimizing anything, get hard numbers. Here's a curl timing command I use to measure raw connection latency to both APIs:

# Measure connection phases to Anthropic
curl -o /dev/null -s -w "\
DNS lookup:      %{time_namelookup}s\n\
TCP connect:     %{time_connect}s\n\
TLS handshake:   %{time_appconnect}s\n\
First byte:      %{time_starttransfer}s\n\
Total:           %{time_total}s\n" \
https://api.anthropic.com/v1/models

# Same for OpenAI
curl -o /dev/null -s -w "\
DNS lookup:      %{time_namelookup}s\n\
TCP connect:     %{time_connect}s\n\
TLS handshake:   %{time_appconnect}s\n\
First byte:      %{time_starttransfer}s\n\
Total:           %{time_total}s\n" \
https://api.openai.com/v1/models

From Tokyo, my typical output looks like:

DNS lookup:      0.028s
TCP connect:     0.187s
TLS handshake:   0.412s
First byte:      0.843s

From a US-East server, the same command returns 0.041s to first byte. That's a 20x difference on connection setup alone.

Measuring Streaming First-Token Latency in Python

The curl test measures connection overhead, but for streaming AI APIs, what users actually feel is time-to-first-token (TTFT). Here's a Python snippet that measures this precisely:

import time
import anthropic

def measure_ttft(prompt: str) -> dict:
    client = anthropic.Anthropic()

    start = time.perf_counter()
    first_token_time = None
    token_count = 0

    with client.messages.stream(
        model="claude-3-5-sonnet-20241022",
        max_tokens=200,
        messages=[{"role": "user", "content": prompt}],
    ) as stream:
        for text in stream.text_stream:
            if first_token_time is None:
                first_token_time = time.perf_counter()
            token_count += 1

    end = time.perf_counter()

    ttft = (first_token_time - start) * 1000
    total = (end - start) * 1000

    return {
        "ttft_ms": round(ttft, 1),
        "total_ms": round(total, 1),
        "tokens": token_count,
        "throughput_tps": round(token_count / ((end - first_token_time)), 1),
    }

if __name__ == "__main__":
    result = measure_ttft("Explain TCP slow start in two sentences.")
    print(f"Time to first token: {result['ttft_ms']}ms")
    print(f"Total time:          {result['total_ms']}ms")
    print(f"Throughput:          {result['throughput_tps']} tokens/sec")

Run this 5–10 times and average the results. On a cold connection from Southeast Asia, I consistently measured 780–920ms TTFT. That's the number we want to crush.

The Fix: Route Through Infrastructure Close to the API

The insight is simple: if the AI APIs live in the US, your traffic should enter the US network as close to those endpoints as possible — not traverse 15 BGP hops across the Pacific first.

There are a few approaches:

Option 1: Deploy your backend in the US. If you control your server, move it to us-east-1. This is the cleanest solution but not always feasible — your users might be in Asia, your data residency requirements might be regional, or you might be running on a local machine during development.

Option 2: Use a regional proxy or accelerator. Route your AI API traffic through an optimized path that has a PoP (point of presence) near the Anthropic/OpenAI data centers. The proxy handles the long-haul routing on an optimized backbone, and your server only needs to reach the nearest proxy node.

This is where I found TonBoVPN genuinely useful. It's designed specifically for routing AI API traffic — you set HTTPS_PROXY in your environment, and your Claude/OpenAI calls get routed through nodes with optimized paths to US API endpoints. The setup is literally one environment variable:

export HTTPS_PROXY=http://your-tonbovpn-endpoint:port

# Your existing Python code works unchanged
python your_app.py

Both the anthropic and openai Python SDKs respect standard proxy environment variables, so there's zero code change required.

Real-World Numbers

After switching to proxied routing, here's what my TTFT measurements looked like across different Asian cities (averages over 20 runs each):

Location	Direct TTFT	Proxied TTFT	Improvement
Tokyo	820ms	195ms	4.2× faster
Seoul	760ms	170ms	4.5× faster
Singapore	690ms	155ms	4.5× faster
Sydney	950ms	220ms	4.3× faster

The 3–4× improvement is consistent across regions. More importantly, the variance dropped dramatically — direct calls would sometimes spike to 2,000ms during peak hours; proxied calls stayed under 300ms at P95.

Why This Matters for Streaming UX

For non-streaming API calls, latency is just latency — your user waits, the response arrives. But for streaming, TTFT is the difference between a UI that feels alive and one that feels broken.

Human perception research puts the "feels instant" threshold at around 100ms and "noticeable delay" at 300ms. At 800ms TTFT, users genuinely think the app is loading or broken. At 180ms, the first token appears before they've consciously registered waiting.

If you're building any kind of chat interface, code assistant, or real-time AI feature for users outside the US, optimizing TTFT isn't a nice-to-have — it's the single highest-leverage UX improvement you can make.

Quick Checklist

[ ] Run the curl timing test against both API endpoints from your actual server location
[ ] Measure baseline TTFT with the Python snippet above
[ ] If TTFT > 300ms, routing is your bottleneck (not the model)
[ ] Try proxied routing via TonBoVPN or a US-region proxy
[ ] Re-run measurements and compare P50 and P95 (variance matters as much as average)
[ ] If deploying to production, instrument TTFT as a metric in your observability stack

Conclusion

Slow AI API responses from outside the US are almost always a routing problem, not a model problem. The fix is straightforward: get your traffic onto an optimized path to US infrastructure as early as possible, whether that's moving your server, using a regional accelerator, or proxying through a service built for this use case. Measure first, optimize second — the curl and Python snippets above give you everything you need to quantify the problem and verify the fix.

How to Fix AI API Failures That Look Like Rate Limits but Are Actually Network Issues

cauqjbwkerl — Tue, 30 Jun 2026 07:01:15 +0000

TL;DR

If your OpenAI, Claude, or Gemini API calls are failing with cryptic errors that look like rate limits, the real culprit is often your network — ISP routing, DNS pollution, or TCP RST injection. A real 429 has a JSON body and a Retry-After header; a network failure gives you an empty response, a connection reset, or a timeout. Here's how to tell them apart and fix it systematically.

I spent two frustrating days last month convinced I'd somehow blown through my OpenAI quota. My Python script kept dying with RateLimitError, but the OpenAI dashboard showed I'd barely touched my limits. Sound familiar? If you're working from Southeast Asia, mainland China, or certain parts of the Middle East, this is a surprisingly common trap.

Let me walk you through exactly how I diagnosed it and what I did to fix it.

Real 429 vs. Network Failure — Know the Difference

This is the first thing to nail down, because the fix is completely different depending on which one you're dealing with.

A genuine rate limit (HTTP 429) always has:

An HTTP status code of 429 in the response
A Retry-After header telling you how long to wait
A JSON body like {"error": {"type": "rate_limit_error", "message": "..."}}

A network-level failure looks like one or more of these:

ConnectionResetError or ConnectionRefusedError
requests.exceptions.ConnectionError with an empty response body
TimeoutError or ReadTimeout with no HTTP status at all
The Python OpenAI SDK raising APIConnectionError instead of RateLimitError

The SDK wraps both into similar-looking exceptions, which is why they're easy to confuse. The key is to look at the exception class and the response body, not just the error message string.

Step 1: Verbose curl to the API Endpoint

Before touching any code, go raw. Run this from your terminal:

curl -v --max-time 15 \
  -H "Authorization: Bearer $OPENAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"ping"}],"max_tokens":5}' \
  https://api.openai.com/v1/chat/completions 2>&1 | head -80

Watch the output carefully:

If you see * Connected to api.openai.com followed by TLS handshake lines and then an HTTP response — your network path is basically working.
If you see * Connection reset by peer or curl: (35) OpenSSL SSL_connect — you have a network-level block, likely TCP RST injection.
If it just hangs until timeout — routing issue or DNS resolution is hitting a poisoned record.

Do the same for Anthropic:

curl -v --max-time 15 https://api.anthropic.com/v1/messages \
  -H "x-api-key: $ANTHROPIC_API_KEY" \
  -H "anthropic-version: 2023-06-01" \
  -H "content-type: application/json" \
  -d '{"model":"claude-haiku-20240307","max_tokens":5,"messages":[{"role":"user","content":"ping"}]}' 2>&1 | head -80

Step 2: Check DNS Resolution

DNS pollution is a real thing in several regions. Your ISP may be returning a bogus IP for api.openai.com.

# What does your default resolver say?
nslookup api.openai.com

# Compare against a clean resolver
nslookup api.openai.com 8.8.8.8
nslookup api.openai.com 1.1.1.1

If the IPs are different — especially if your default resolver returns a private IP range or a local redirect — you've found your DNS problem.

Step 3: Traceroute to See Where Traffic Dies

# Linux/macOS
traceroute -n api.openai.com

# Windows
tracert api.openai.com

If the trace stops at a hop inside your ISP's network (typically hops 3–8) and never reaches the destination, that's ISP-level routing interference. If it reaches international exchange points but then drops, it's a peering or transit issue.

Step 4: Enable Python SDK Debug Logging

Once you suspect a network issue, enable verbose logging in the OpenAI Python SDK to see exactly what's happening at the HTTP layer:

import logging
import httpx
import openai

# Enable full HTTP request/response logging
logging.basicConfig(level=logging.DEBUG)
logging.getLogger("httpx").setLevel(logging.DEBUG)
logging.getLogger("openai").setLevel(logging.DEBUG)

client = openai.OpenAI()

try:
    response = client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": "hello"}],
        max_tokens=10
    )
    print(response.choices[0].message.content)
except openai.APIConnectionError as e:
    print(f"Network-level failure (not a rate limit): {e}")
except openai.RateLimitError as e:
    print(f"Genuine rate limit: {e}")
except openai.APIStatusError as e:
    print(f"API error {e.status_code}: {e.message}")

The APIConnectionError vs RateLimitError distinction is your smoking gun.

The Fix: Route Through a Reliable Tunnel

Once I confirmed it was a network issue (my traceroute was dying at hop 6 inside my ISP), the solution was to route API traffic through a tunnel with better international connectivity.

You have a few options:

Configure your system proxy if you already have a VPN or proxy running locally
Use a purpose-built tunnel optimized for this kind of traffic

For option 1, here's how to configure the proxy in both Python and Node.js:

Python (OpenAI SDK):

import httpx
import openai

# If your local proxy is running on port 7890
proxy_url = "http://127.0.0.1:7890"

client = openai.OpenAI(
    http_client=httpx.Client(
        proxy=proxy_url,
        timeout=30.0
    )
)

response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[{"role": "user", "content": "hello"}]
)
print(response.choices[0].message.content)

Or via environment variable (works for most SDKs):

export HTTPS_PROXY="http://127.0.0.1:7890"
export HTTP_PROXY="http://127.0.0.1:7890"
python your_script.py

Node.js (using the official OpenAI package):

import OpenAI from 'openai';
import { HttpsProxyAgent } from 'https-proxy-agent';

const proxyAgent = new HttpsProxyAgent('http://127.0.0.1:7890');

const client = new OpenAI({
  httpAgent: proxyAgent,
});

const response = await client.chat.completions.create({
  model: 'gpt-4o-mini',
  messages: [{ role: 'user', content: 'hello' }],
});
console.log(response.choices[0].message.content);

For option 2, after trying a few generic VPN services that added latency or had unreliable uptime, I ended up using TonBoVPN, which is specifically tuned for AI API traffic. The difference in latency and connection stability to api.openai.com and api.anthropic.com was noticeable — it also handles the DNS resolution cleanly, which matters if your ISP is doing DNS poisoning. You still configure it the same way via the local proxy port.

Putting It All Together: A Diagnostic Checklist

Here's the exact order I run through now whenever an AI API starts misbehaving:

Check the exception type — APIConnectionError = network, RateLimitError = real quota
Check your dashboard — OpenAI, Anthropic, and Google all show real-time usage
Run verbose curl to the endpoint and look for TLS handshake vs connection reset
Compare DNS resolution between your default resolver and 8.8.8.8
Run traceroute and see where packets stop
Enable SDK debug logging for the full HTTP picture
Configure a proxy and test again

One more thing: if you're building a production service that serves users in regions with connectivity issues, consider implementing a retry wrapper that distinguishes between these error types. Retrying a genuine rate limit with exponential backoff makes sense. Retrying a TCP RST injection 10 times in a row just wastes time and quota.

import time
import openai

def call_with_smart_retry(client, max_retries=3, **kwargs):
    for attempt in range(max_retries):
        try:
            return client.chat.completions.create(**kwargs)
        except openai.RateLimitError as e:
            wait = int(e.response.headers.get("retry-after", 60))
            print(f"Rate limited. Waiting {wait}s...")
            time.sleep(wait)
        except openai.APIConnectionError as e:
            print(f"Connection error on attempt {attempt+1}: {e}")
            if attempt < max_retries - 1:
                time.sleep(2 ** attempt)  # brief backoff, then check your tunnel
            else:
                raise
    raise RuntimeError("Max retries exceeded")

Conclusion

Most "rate limit" errors I've seen from developers in Asia are actually network failures in disguise — and the fix is completely different from what you'd do for a real 429. The diagnostic flow (curl, DNS check, traceroute, SDK logging) takes about five minutes and tells you exactly where the problem lives. Once you know it's a routing or DNS issue, configuring an HTTPS_PROXY in your SDK is a one-liner that usually solves it immediately. Don't spend hours tweaking retry logic or downgrading your API tier when the problem is three hops into your ISP's network.