Batch scraping fails quietly. A crawler that loses 4% of its requests retries on the next run and nobody notices. Real-time retrieval-augmented generation is the opposite: the fetch happens inside the user's request, on the critical path between "they asked a question" and "the answer rendered." Every second your egress spends connecting through a half-dead proxy is a second the human is watching a spinner. And when the exit is blocked outright, you do not get a degraded dataset — you get a confident hallucination, because the retriever returned nothing and the generator filled the silence with plausible fiction.
The thesis: the reliability and tail-latency properties of your proxy exits are a first-order part of your RAG SLO, not an infra footnote. Batch collection lets you optimize for throughput per dollar; live retrieval inverts every one of those priorities. You must engineer the egress layer for availability and bounded latency, and you must treat a slow exit and a blocked exit as the same class of incident — both are user-facing failures.
The budget is small and unforgiving
A user-facing retrieval call has maybe 1.5 to 3 seconds end-to-end before you should give up and serve a degraded answer with a stale cache or a disclosure. Subtract embedding the query, a vector search, and the LLM's own token time, and the network portion for fetching live sources is often a few hundred milliseconds. A residential proxy exit adds a hop: a TCP connection plus TLS plus the relay through the provider, then the origin's own response time. Median is usually fine. It is the tail that kills you — one exit in a pool of fifty that takes 4 seconds to establish, or black-holes SYNs until your 2-second timeout, will decide your p99 single-handedly.
Batch pipelines hide the tail by retrying and moving on. In live RAG there is no "move on"; the tail is the latency your user experiences. So the exit pool is not "somewhere we send requests," it is a distributed system component with an availability objective and a p99 budget, exactly like a database replica or a cache.
A health-checked, circuit-broken exit registry
The naive design — pick a random exit per query and hope — fails precisely when it matters, because the exits that get blocked do so non-randomly: certain ASNs get soft-blocked by a given origin during certain hours, and those are exactly the ones your naive round-robin keeps handing to live traffic. You need to observe outcomes and pull failing exits out of rotation automatically, with a per-exit circuit breaker, while keeping enough healthy exits to serve a burst.
import threading
import time
from dataclasses import dataclass, field
@dataclass
class ExitState:
ip: str
window: list = field(default_factory=list) # (monotonic_time, ok: bool, latency_s)
open_until: float = 0.0
capacity: int = 8 # max in-flight per exit
class ExitRegistry:
"""Tracks per-exit success and latency, trips a circuit breaker on a bad
exit, and only hands out exits that are currently closed and under capacity."""
def __init__(self, exits: list[str], window_s: float = 30.0,
min_ok_rate: float = 0.6, trip_s: float = 20.0,
max_inflight: int = 8):
self._states = {ip: ExitState(ip=ip, capacity=max_inflight) for ip in exits}
self._lock = threading.Lock()
self.window_s = window_s
self.min_ok_rate = min_ok_rate
self.trip_s = trip_s
def _prune(self, st: ExitState, now: float):
st.window = [rec for rec in st.window if now - rec[0] <= self.window_s]
def choose(self, now: float | None = None) -> str:
now = now or time.monotonic()
with self._lock:
ready = []
for st in self._states.values():
if now < st.open_until:
continue
self._prune(st, now)
ok = sum(1 for _t, o, _l in st.window if o)
total = len(st.window)
if total >= 5 and (ok / total) < self.min_ok_rate:
st.open_until = now + self.trip_s # trip the breaker
continue
if in_flight(st) < st.capacity:
ready.append(st)
if not ready:
# Every exit tripped: fall back to the soonest-to-recover rather
# than fail the user outright. Better a slow answer than a
# hallucinated one, but log it loudly.
fallback = min(self._states.values(), key=lambda s: s.open_until)
return fallback.ip
return min(ready, key=self._ewma_latency).ip
def record(self, ip: str, ok: bool, latency_s: float, now: float | None = None):
now = now or time.monotonic()
with self._lock:
self._states[ip].window.append((now, ok, latency_s))
def _ewma_latency(self, st: ExitState) -> float:
lats = [l for _t, _o, l in st.window if l is not None]
return sum(sorted(lats)[-max(1, len(lats)//5):]) / max(1, min(5, len(lats))) or 0.0
choose() prefers exits with a good recent p-tail latency, not just a good mean, because the tail is what busts the SLO. record() is called on every completion, so a start to fail under a specific origin feeds straight into opening its circuit. And the not ready branch — falling back to the soonest-to-recover exit — is the difference between serving a slow answer and serving an empty retrieval that the model papered over. When every exit is tripped, that is your paging event, not a silent retry.
The per-query deadline that actually matters
The registry helps you pick well; it does not make you fast. For that you need a deadline that spans the entire retrieval, plus a hedge: fire the fetch through your best exit, and if it has not returned by roughly half the remaining budget, fire a second attempt through the next-best exit and take whichever wins. Hedged requests turn a p99 tail into a p50-ish experience at the cost of a little redundant proxy GB — a trade you make willingly on the user's critical path and never on the batch path.
import asyncio
async def retrieve_with_hedge(registry, urls, deadline_s: float, fetch):
loop = asyncio.get_running_loop()
budget = loop.time() + deadline_s
urls = list(urls)
in_flight = set()
async def one(url):
ip = registry.choose()
try:
remaining = budget - loop.time()
if remaining <= 0:
return None
t0 = loop.time()
res = await asyncio.wait_for(
fetch(url, proxy=ip),
timeout=max(0.1, remaining),
)
registry.record(ip, ok=True, latency_s=loop.time() - t0)
return res
except (asyncio.TimeoutError, Exception):
registry.record(ip, ok=False, latency_s=deadline_s)
return None
tasks = {asyncio.create_task(one(urls[0]))}
hedge_at = loop.time() + deadline_s * 0.5
while tasks:
now = loop.time()
if len(tasks) == 1 and urls[1:] and now >= hedge_at:
tasks.add(asyncio.create_task(one(urls[1])))
done, tasks = await asyncio.wait(
tasks, timeout=max(0.05, budget - now),
return_when=asyncio.FIRST_COMPLETED,
)
for t in done:
val = t.result()
if val is not None:
for other in tasks:
other.cancel()
return val
return None # caller MUST mark this degraded and prefer cached context
The line return None is the most important one in the file. When retrieval fails, RAG must not proceed as if it succeeded. Return a status to the generator: "live fetch failed, answer from cache only and say so." A model that knows it has no fresh context will hedge its claims; a model that silently got an empty tool result will not. Distinguish "no relevant content" from "the egress was down," because to the user those must not look the same.
Warm the exits before you need them
A cold TCP+TLS handshake to a residential relay you have not touched in ten minutes can cost hundreds of milliseconds you do not have in the budget. Keep the hot exits warm with a low-rate background prober that also feeds the registry: connect, fetch a tiny known-stable URL, record the latency, repeat. This doubles as your health check — an exit the prober cannot reach is exactly the exit you must not hand to a live query. For a RAG system doing, say, two retrievals per second, a prober that keeps twenty exits warm and fresh costs almost nothing and buys you the tail you cannot afford.
The honest version
Live RAG moves the proxy layer from the background to the critical path, and that inverts your design goals. Throughput-per-dollar stops mattering; bounded p99 latency and availability start mattering. Model your exit pool as a distributed dependency with a circuit breaker per exit and a fallback that never serves an empty retrieval as success. Hedge the second attempt at half-budget so one slow relay does not define your tail. And warm your exits before users arrive, because the first request through a cold relay is always the one you can least afford. A batch scraper can retry tomorrow. Your user cannot, and if your egress is not engineered for that fact, the failure will show up as a confident wrong answer in someone's chat window — and no amount of prompt engineering will trace it back to a dead exit in a pool of fifty.
Disclosure: I use Thordata's residential proxies for this project. New users get 500MB free — code thor020 (10% off): https://www.thordata.com/?ls=uXcSHJzx&lk=02-tele
Top comments (0)