The Death of Transient Memory: Engineering a Zero-Cost B2B LLM Edge Cluster

A functional local inference node is merely a prototype. An observable, stateful inference node is enterprise infrastructure.

The current standard of wrapping quantized LLMs in basic FastAPI endpoints and exposing them to the global internet is fundamentally flawed. When subjected to concurrent B2B payloads, in-memory token buckets fracture. The ASGI event loop bottlenecks, GPU VRAM fragments, and the node dies silently.

To eradicate transient memory anomalies and bypass hyperscaler billing, I engineered a fully distributed, zero-trust Docker bridge matrix.

Here is the strict architectural progression of the edge cluster.

1. The Compute Layer (Bypassing VRAM Fragmentation)

Standard localized nodes load unoptimized FP32 tensors, instantly saturating consumer hardware.
This matrix utilizes google/flan-t5-base with 8-bit precision (BitsAndBytesConfig). To allow enterprise-specific instruction alignment without full-parameter overhead, the base model is merged with a Low-Rank Adaptation (LoRA) via peft.

2. The State Layer (Eradicating Localized Memory)

FastAPI dict objects cannot manage concurrent state. We strictly externalize it.

• Authorization: API keys are validated against a persistent PostgreSQL volume using asynchronous non-blocking I/O (asyncpg).
• Atomic Rate Limiting: A localized Redis container executes asynchronous Lua pipelines (transaction=True). This guarantees atomic evaluations of payload frequency, violently returning HTTP 429s to hostile actors before they penetrate the inference queue.

3. The Routing Layer (Zero-Trust Isolation)

The application layer is entirely severed from the localized host environment to prevent kernel port collisions.
All internal traffic routes through an isolated Traefik reverse proxy. Global ingress is handled via a direct HTTP2 TCP Cloudflare tunnel, bypassing hypervisor UDP limits and local firewall ACLs entirely.

4. The Observability Matrix

Infrastructure without telemetry is a black box.
Prometheus silently scrapes the Uvicorn workers every 5 seconds. The TSDB is explicitly whitelisted from the Redis token bucket to prevent a self-inflicted denial of service. Grafana is provisioned via Infrastructure as Code (IaC), etching the dashboards directly into the container state.

The Chaos Engineering Benchmark

To mathematically prove the architecture's load-bearing capability, the node was subjected to a 150-concurrent-user synthetic swarm utilizing Locust.

The telemetry proves the mathematical truth:

The atomic Redis transactions successfully identified the payload overflow in milliseconds, aggressively returning HTTP 429s. The Uvicorn workers remained shielded, and the p95 latency for accepted trans-continental payloads remained perfectly stable.

The complete, verifiable infrastructure is open-sourced here:
GitHub Repository Link

Question for the infrastructure architects: When load-balancing edge inference, are you standardizing on Traefik or native Nginx for your internal Docker DNS resolution? Defend your routing latency below.