Articles
🔍 AI Gateway: Cost Control, Failover, and New Risk
A LiteLLM AI gateway design that turns token spend and provider incidents into controlled failures. It covers virtual key minting with per-key max_budget (over-budget returns 400 while other keys keep working), usage-based routing with declared fallbacks and retries, Redis-backed response caching validation, and Prometheus/OTel spend attribution via /metrics/. The closing section names the new risk: the gateway becomes the critical path, so version pinning by image digest and Kubernetes liveness/readiness are mandatory.
🔍 APISIX Throughput Regression: Beyond the Flame Graph
An investigation into attributing a regression whose costs are obscured across shared paths, interpreter dispatch, JIT traces, allocation, and caller frames. It combines 500 Hz eBPF profiling with request-phase call counting, LuaJIT jit.attach correlation of trace start/stop/abort events, jit.dump validation, and paired A/B tests. The customized APISIX build revealed nine Global Rule scans over 100+ plugins per request and a 43.1% normalized throughput gap, motivating cached filtering and elimination of disabled-feature work.
🔍 Building a Wallet Event Backbone with NATS JetStream: Lessons from BroSettlement's Staging Tests
A JetStream wallet backbone: a PostgreSQL transactional outbox commits business rows plus an outbox event in one TX, then a worker publishes to JetStream only after commit to avoid rollback and lost-event failures. The design targets at-least-once with dedup by global eventId and chain transaction identity, makes HTTP retries safe with Idempotency-Key, and isolates internal JetStream consumers from a filtered permissioned public WebSocket stream. TRON Nile staging produced 926 lifecycle events spanning API 201/503/500 and drives next-step durable-consumer resume/cursor hardening.
🔍 Designing Agent State Machines for Long-Running Business Workflows
A durable, platform-agnostic "agent state machine" blueprint: distinguish business vs deployment lifetime, persist state outside the context window, and wrap every LLM/external read as a journaled durable step to avoid double side effects on replay. It models human approvals as explicit waiting states (Temporal signals, LangGraph interrupt/resume, Step Functions callback task with HeartbeatSeconds), adds timeout destinations for escalation, and handles non-atomic resumption by revalidating external state on entry and routing stale decisions to pending_approval.
🔍 Filter Ordering Is the Whole Game: Building an LLM Gateway on Spring Cloud Gateway
An SCG-based LLM gateway that enforces token quotas by reserving an upfront estimate (prompt_chars/chars_per_token + max_tokens) and refunding via post-response reconciliation. It meters by observing usage using a bounded 8KB rolling tail with non-advancing DataBuffer.toString, brace-balanced parsing for nested usage details, and injection of stream_options.include_usage (plus Content-Length fixes). Correctness hinges on strict filter ordering and re-subscribable request bodies to avoid zero-metering and reactive traps like Mono<Void> switchIfEmpty.
🔍 From Ingress to Inference Gateway: Gateway API and the Inference Extension
A Gateway API inference routing pattern: HTTPRoute points to an InferencePool, and Envoy ext-proc delegates model selection to llm-d-router. The EPP buffers the full request, parses OpenAI/Anthropic/vLLM bodies, applies InferenceModelRewrite, runs a plugin dependency graph to score endpoints using queue, KV-cache utilization, and prefix-cache (or precise KV events), then returns the chosen pod to Envoy. It adds saturation-aware 429 admission, body-based routing via X-Gateway-Model-Name, and LoRA-affinity scheduling from lora_requests_info.
🔍 gRPC over a Unix socket, not HTTP: a real IPC tradeoff from a HIPAA-postured edge system
A production ADR that picks client-streaming gRPC over a Unix domain socket for a single-host Go-to-Python DSP hop. It explains why REST/HTTP/JSON fails to match the batch/windowed streaming shape (Send + CloseAndRecv semantics), quantifies latency pressure (10ms/frame, 1000 frames per ~100ms batch) favoring protobuf decode over JSON, and argues for a smaller exposure surface using a filesystem-scoped UDS path; it explicitly limits the decision to co-located processes and mandates TCP gRPC with mTLS if the DSP moves off-host.
🔍 How and Why Netflix Built a Real-Time Distributed Graph: Part 3 — Querying the graph with gRPC…
A practical gRPC execution API for multi-hop real-time distributed graph queries that packages traversal into one request. It uses breadth-first level expansion, adjacency-list node lookup plus streamed fan-out in bounded batches, and async-first orchestration across dedicated thread pools with adaptive concurrency limiting (backoff on timeouts). Results are reduced via hierarchical per-hop filtering (time windows, LATEST vs ANY) and selective EVCache node caching (70-80% hits), with opt-in fail-open enrichment layering.
🔍 How Cloudflare detects MCP traffic and helps secure it
Cloudflare Gateway classifies MCP by the presence of MCP-Protocol-Version and exposes experimental.is_mcp for policy decisions, avoiding URL/hostname heuristics. The article explains how stateless MCP 2026-07-28 moves MCP-Method and MCP-Name onto every request, enabling per-request classification on TLS-inspected paths. It further ties MCP Portal routing to Traffic Source mcp_portal so HTTP/network policies can block detected MCP that bypasses approved portals while preserving permitted Portal traffic.
🔍 OpenAPI vs MCP vs Context Plugins: Benchmarking API Integrations
A reproducible benchmark comparing OpenAPI, Docs MCP, and Context Plugins for a real subscription-billing integration in eShopOnWeb. The study fixes the host app and task, then varies only the provider resource across five runs/condition and grades each run on a strict 24-check readiness gate spanning typed error hygiene, resilience to 503/429, secure config behavior, and drift handling. Context Plugin clears 23-24/24 versus 12-20, with root causes including generated-client transport bypass and deterministic Docs truncation at 29,243 chars.
🔍 Publishing a versioned API contract you can actually trust
An operational trust pipeline for OpenAPI contracts rather than generic versioning advice. It combines CI regeneration and untracked-file checks with clean-tree packaging, Git commit-byte hashing, semantic-version validation, per-artifact SHA-256 manifests, immutable commit-pinned URLs, independently tagged contract releases, and post-publish anonymous verification. It also codifies compatibility details including additive-field semantics, numeric precision, ETag If-Match handling with 428/412 outcomes, and retry-scoped idempotency.
🔍 Reading Load Test Results with Distributed Tracing
A cross-layer "attribution rule" plus clock-consistent latency decomposition for load-test debugging. It shows how to instrument a gateway with an OpenTelemetryPlugin, propagate W3C traceparent from k6 (tempo jslib), then read trace waterfalls to attribute 5xx to the segment between the last failing and first non-failing viewpoint. It also covers measurement artifacts from two-clock subtraction and models tracing cost via head sampling ratio and error-span forced export.
🔍 REST vs gRPC vs GraphQL in NestJS: What the Numbers Actually Show
A reproducible transport benchmark for the NestJS stack (Node 22, localhost) over identical 100 and 2000 record datasets. It benchmarks plain Express, graphql-yoga and @grpc/grpc-js directly rather than through NestJS, whose layer adds a roughly constant cost, then shows the idiomatic NestJS wiring for each. It times gRPC via concurrent worker loops and computes JSON vs Protobuf payload byte sizes. Results show REST ~2.8-3.1x higher req/s than gRPC, explain why grpc-js pure-JS plus localhost erases network/HTTP2 gains, and attribute GraphQL's biggest benefit to requesting only id and name.
AAM moves least privilege from "trusted principal" to the task execution graph: dispatch mints short-lived sender-constrained, task-scoped credentials, then enforces authorization at harness tool-call and network egress boundaries. A stateful Trust Ratchet narrows capabilities on protected events and buffers responses until all enforcement components ACK the new state, failing closed. The Agent Activity Log feeds a Grant Review Loop to adjust future task templates using unused grants and denial evidence.
🔍 The API layer that was fake all the way down
A side-by-side experiment showing that when generated clients never call fetch/http, resilience code is largely untested. The author uses grep to prove absence, then measures 900 mock calls with only setTimeout latency and zero failures. Reusing the same wrapper (250ms timeout, 3 linear-backoff attempts, 6-branch taxonomy) over a real fetch server reveals misclassification: ECONNREFUSED appears as TypeError message "fetch failed" with code only in err.cause.code, so retries are disabled (attempts=1) and timeouts do not cancel in-flight requests.
🔍 The Outbox Pattern Is Not Enough
Operationalizes the transactional outbox by quantifying the hidden freshness capacity from scheduler config, then translating that into SLO and Prometheus alerting. It derives publish ceiling as findTop20 batchSize times fixedDelay (20/5s=4 ev/s), measures oldest pending age peak (191s) from a finite burst to calibrate the alert's for: window, and flags a silent failure mode where FAILED rows stop updating age/backlog gauges. It recommends an unconditional OutboxPublishTerminalFailure alert on outbox_failed with runbook-driven re-drive semantics.
🔍 Your agent should not be as powerful as the person who asked: the second ceiling most MCP setups…
The intersection rule for MCP agent authority, plus a diagnosis of why hosted MCP can eliminate the action-scope ceiling, with Notion DCR as the worked case. The article details a two-point enforcement design: discovery-time allowlisting of exact read tools via tools/list, and execution-time pre-tools/call policy checks (fail closed for unknown tools). It connects July 2026 MCP RC routing changes (per-request metadata, Mcp-Method/Mcp-Name) to making proxy enforcement cheaper, and gives a decision procedure for choosing where ceiling 2 should live.
AWS
🔍 Credential Brokering Patterns for AI Agents Part 2: Protecting the Token Vault with AWS KMS
A threat-model-driven comparison of KEK custody modes for an agent credential token vault, ending in broker-gated KMS grants. The article uses envelope AES-256-GCM with per-write DEKs and row-bound AAD/EncryptionContext, then shows Mode 2's auditable but still standing kms:Decrypt posture vs Mode 3 where a separate broker (CreateGrant) re-verifies the subject token and issues short-lived Decrypt grants with EncryptionContextSubset={owner, resource, elicitation_id}, preventing controller-only mass decrypt and making decrypt per-request and revocable via KMS/CloudTrail.
🔍 DynamoDB Streams is not an outbox
Reframes "Streams as outbox" as missing durable intent and proposes a transactional outbox in DynamoDB. It shows using TransactWriteItems to atomically update mutated state and insert a PENDING outbox item (ULID eventId, idempotencyKey, GSI-backed pendingBucket/pendingAt). Downstream Lambdas publish from outbox records, relying on idempotency and marking PUBLISHED with TTL, while acknowledging Streams' 24h retention, per-shard ordering, and implementing a scheduled reconciler to republish anything older than the retry window.
Apache Camel
🔍 Camel Routes as AI Tools: Unified Tooling and MCP Server in Camel 4.22
The design for Camel 4.22: ai-tool routes register into a centralized AiToolRegistry with tags, letting LangChain4j and Spring AI bridges translate AiToolSpec to each framework's tool/function format without duplicate definitions. A separate camel-mcp-server-api layer watches the registry via McpServerBridge and exposes matching tagged tools over MCP, with tag-based filtering as a security boundary, SPI-based engines per runtime (Vert.x, Quarkus, Spring AI MCP), and built-in protections like timeout, error sanitization, and name collision checks.
Apache Kafka
🔍 Configuring Apache Kafka for High-Load Systems
A metric-driven Kafka 3.x-4.x tuning runbook: it ties broker saturation (NetworkProcessorAvgIdlePercent, RequestHandlerAvgIdlePercent, queued.max.requests) and replication limits (num.replica.fetchers, UnderReplicatedPartitions) to producer batching and reliability (acks=all with idempotence, per-partition batch.size, linger.ms, buffer.memory sizing, max.in.flight hard cap) and consumer latency/rebalance stability (fetch.min.bytes, max.partition.fetch.bytes sizing, max.poll.interval.ms vs session.timeout.ms, CooperativeStickyAssignor and static membership).
🔍 Consumer Resilience in Event-Driven Architecture
A consumer resilience blueprint that treats idempotency safety net, staged retry, and DLQ as one design. It details Spring Kafka retry routing where transient exceptions go to retry topics with exponential backoff plus jitter, while DLQ envelopes preserve failureCategory/attemptCount/context. The central move is "one retry topic per tier" to align Kafka offset order with message readiness using retry-not-before header, consumer pause/resume, and an integrity-conflict log for same messageId with different content-hashes.
🔍 How We Kept OTP Messages Fast While Processing 301 Million Kafka Events a Month
An architecture for Kafka prioritization without native priority queues: isolate traffic by priority and channel into separate topics/consumer groups, tune per-priority batching via max.poll.records, then enforce strict P1->P2->P3 scheduling where capacity is scarce using a shared gate concept. It details the delayed-release gate (versioned short hold after P1 completion) that closes a micro-window race, discusses local vs cluster-wide coordination options, and specifies what to monitor (Kafka lag, provider permits, gate blocked/delay, starvation).
🔍 Kafka Is Free Until It Isn’t : INBROAD
A Kafka TCO anatomy tied to specific cost levers. It quantifies cross-AZ amplification (e.g., follower fetching via KIP-392, replication factor effects) and shows how partition count drives file-descriptor, controller metadata, leader election, rebalance, and page-cache pressure. It also covers cost restructures with managed services and tiered storage (KIP-405) plus producer, broker, and JVM tuning knobs to bend the cost curve.
🔍 Optimizing Kafka with Tiered Storage
A production Kafka Tiered Storage case study: DT wires an RSM plugin (Aiven Open) so when segments roll, they get chunked and uploaded to GCS; RLMM updates `remote_log_metadata` and brokers fetch remote chunks into a cache on demand for older offsets. The article gives the configs that matter (remote.log.storage.enable, remote.storage.enable, local.retention.ms, retention.ms) and a tuning tradeoff for log.segment.bytes (1GiB to 200MB/50MB) plus cache-disk sizing to avoid self-eviction, reporting over 70% cost reduction.
🔍 Taming Kafka Lag Spikes with KEDA Scale-to-Zero
Combines lag-driven scale-to-zero with per-pod drain-rate engineering, validated in a runnable lab: a KEDA ScaledObject scales StatefulSet from minReplicaCount=0 using Kafka consumer-group lag, then throughput is fixed by tuning max.poll.records and fetch.min.bytes and performing bulk idempotent upserts per poll. Scaling stays stable with CooperativeStickyAssignor and static membership (group.instance.id via StatefulSet pod name) plus coarse scale-up and cooldown so rebalances don't pause consumption.
🔍 The backlog that killed my Kafka consumer
The failure-mode math behind a "deadline vs unbounded loop" pattern in Kafka consumers. The author explains how draining a 500k TTL-expiry backlog in one pass blocks poll long enough to exceed max.poll.interval.ms (default 300s), triggering partition revocation/rebalance cascades across replicas. The fix is to bound eviction work per iteration (evict N, process, commit, return to poll) and size N against worst-case per-item cost to stay under the poll deadline, which also adaptively widens the effective dedup window under load.
🔍 The Classloader That Killed Exactly One Kafka Producer
A production postmortem showing how Spring Boot LaunchedClassLoader isolation breaks Kafka producer creation when it runs on ForkJoin common-pool threads. It proves ForkJoinWorkerThread forces contextCL to System, while kafka-clients 4.1.2 resolves reporter classes reflectively (Class.forName) using contextCL; in fat jars BOOT-INF/lib is invisible to System, so JmxReporter lookup throws ClassNotFoundException. Fix: pre-create all producers at ApplicationReadyEvent so factory caching binds resolution to the startup classloader, avoiding the call-order dependency.
🔍 The Kafka Incident That Was Just a Certificate (And the Operator That Saved MirrorMaker)
A CA-rotation integration pattern for Strimzi-managed Kafka plus MirrorMaker2 across namespaces: Strimzi renews cluster-ca/clients-ca in the source secret, Reflector copies it using reflector.v1.k8s.emberstack.com annotations, and the article proves freshness by matching mirrored secret reflected-version with the source secret resourceVersion and timestamps (operator write then reflector at +1s). It also clarifies the division of labor: Reflector distributes only, Strimzi's KafkaRoller triggers MirrorMaker2 restart after the truststore update.
🔍 Who Did This? Identity Across Async Boundaries
An implementation that treats identity as event data: capture IdentityContext on the HTTP request thread, persist actor_username/email/roles plus trace_id inside the same transaction as the outbox row, publish later by explicitly attaching these fields as Kafka headers, and on consume restore identity into MDC/IdentityContext then clear it in a finally block to avoid Virtual Thread leakage. Fault injection proves `actor*` is retained on retries exhausted via @DltHandler dead_letter_events persistence._
🔍 Why committing Kafka offsets out of order loses data
Demonstrates that buffered Kafka consumers cannot "ack" per message; committing after enqueue/receipt loses in-flight buffered records on restart (4,456 in simulation) with no errors. Shows the correct mechanism: carry the partition offset through a FIFO dedup buffer and commit only at eviction time after processing, maintaining the invariant that the committed offset is below all still-buffered offsets. Quantifies the bounded replay cost (20,001) and stresses downstream idempotent sinks.
🔍 Why I Killed Our Kafka Cluster (And What I'd Do Differently)
A Kafka-to-SNS/SQS migration postmortem for payments messaging, detailing how to map Kafka topics to AWS fan-out and queue semantics without transaction impact. It covers per-topic dual-publish with output comparison, consumer-side schema governance to avoid breaking-change deserialization, DLQ-driven operational health, and ordered topics via SQS FIFO message-group IDs (account_id/merchant_id/processor_id) plus throughput modeling and shadow-mode cutovers.
Azure
🔍 Tracking Email Delivery Status with Webhooks: From “Sent” to “Actually Delivered”
Eliminates a SparkPost-to-internal lookup table by embedding a per-email status_update_url inside transmissions metadata (rcpt_meta), then having the webhook handler call that URL with final delivery/bounce reasons. The article models outcome types with a NotDeliveredEventTypes set, accounts for delay being non-final, and uses a Durable Function to offload work so the endpoint responds within SparkPost's 10s window. It also covers securing webhooks via basic auth or OAuth and planning signed correlation links.
Debezium
🔍 The Debezium snapshot event_id trap: why LSN-based idempotency keys silently drop rows
Shows why LSN-derived idempotency keys (commit_lsn/change_lsn/event_serial_no) violate the uniqueness requirement specifically for Debezium snapshot op:r, where change_lsn is null/empty and event_serial_no becomes 0 so thousands of distinct rows share one event_id. The fix is to branch key derivation by mode and use the payload row primary key when isSnapshot, keeping the LSN only for ordering. It includes Spring inbox dedup (@Transactional ProcessedEvent) plus unit tests covering snapshot vs live envelopes and observability for skipped duplicates.
Google Cloud
🔍 We turned off Pub/Sub and nobody noticed
A production migration to active-active dual brokers with a zero-impact Pub/Sub shutdown, backed by chaos testing. An eventadapter abstraction swaps Pub/Sub and NATS at runtime; publishing uses ULID message.ID hashed via fnv plus configurable weighted split, with per-broker circuit breakers to short-circuit failed publishes and retry the other broker. Subscribing uses a single-slot Peek/Receive "Inbox" per broker and a semaphore-bounded scheduler that dispatches the oldest head (OCF-style) for fairness-aware backlog handling.
🔍 When Every API Request Started Returning 429: Debugging SpikeArrest and Quota in Google Cloud…
Diagnoses "nearly all requests 429" in Apigee by showing that SpikeArrest and Quota behavior depends entirely on the resolved Identifier at execution time. It walks through verifying which policy failed, checking <Identifier ref="developer.app.name"/> availability and flow ordering vs VerifyAPIKey/OAuth resolution, and then applying safer fallbacks like <Identifier ref="client.ip"/>. It also contrasts SpikeArrest vs Quota and explains Distributed/Synchronous quota tradeoffs with concrete XML.
Kong
🔍 Intelligent Model Routing: Kong AI Gateway Applies NVIDIA NeMo Switchyard Across Model Traffic
The split of concerns: Kong AI Gateway remains the centralized request-path boundary while NVIDIA NeMo Switchyard runs as an external model selection decision service invoked per request. The architecture returns a model target to Kong, supports a customer fallback when the decision service fails, and applies gateway policies (vaulted credentials, PII masking, token metering, audit logging, semantic caching, multi-provider failover). Benchmarks show tuning stage_router threshold 0.3->0.5 dropping frontier escalations 85%->17% with 43.7% lower cost.
MuleSoft
🔍 Can MuleSoft Omni Gateway Secure an MCP Server That Runs Arbitrary SQL?
A production-oriented example of governing an MCP SQL tool with MuleSoft Omni Gateway, proving that non-auth policies can be applied without modifying server code. The article details registering an MCP asset via Exchange, routing through the gateway with correct base-path mapping, then enabling MCP Support plus Client ID Enforcement, MCP PII Detector, and MCP Payload Optimization. It demonstrates concrete diffs where PHONE_NUMBER is masked and null COMMISSION_PCT fields are stripped from tool responses while calls still succeed (200) with valid credentials.
🔍 Hands-On With MuleSoft Omni Gateway: What I Learned Building a Multi-LLM Demo
Reproducible, integration-level debugging of MuleSoft Omni Gateway multi-LLM routing: (1) Destination URL must be host-only because the Envoy path rewriter appends the provider path internally, otherwise requests 404 with empty bodies; (2) Anthropic and Gemini native request-format policy paths described in docs appear not live on the tenant, yielding gateway-shape rejection errors; (3) the Fallback route only triggers on gateway routing/match failures, not upstream HTTP errors like 401.
Mergers & Acquisitions
🤝 OpenRouter called itself the “Stripe for LLMs” — now Stripe’s swooped in to buy it
Stripe is acquiring OpenRouter for a reported $8 billion, its largest known deal, tying its AI billing plans to OpenRouter's model-gateway architecture: a single developer API that routes across hundreds of models and providers with price, latency, and quality optimization plus automatic failover. The piece frames the technical integration goal as tokenomics, where Stripe's LLM token metering and proxy capabilities can unify economic controls with model-selection logic under one platform.
Releases
APISIX 3.18.0 extends the AI Gateway with ai-cache (Redis exact and semantic L2 using embeddings, plus SSE capture/replay) and ai-proxy-multi semantic balancing that embeds per-instance examples and routes by cosine similarity with fallback. It also switches AI upstream calls to the FFI HTTP client by default, integrates Lakera Guard for prompt-injection and policy enforcement with streaming buffering, and adds AI-focused Prometheus metrics (ttft vs total, cache hit/miss/bypass) plus numerous breaking security defaults (buffer limits, fail-closed/skip modes, OIDC validation).
🚀 Apache Camel 4.22 What's New
Camel 4.22 is LTS, leaving 4.18.x and 4.22.x supported as 4.14.x reaches end of life. The headline is Camel TUI, a terminal app for monitoring and developing integrations across 30+ tabs, with a YAML DSL editor offering Tab completion for EIPs, endpoint options and application.properties keys. The CLI moves from Preview to Stable with one-line installers and doctor diagnostics. Camel AI adds camel-ai-tool and an MCP server, security-by-default continues (JEP-290 filters, URI allow-lists for toD and enrich, JWT hardening), and Splitter gains chunking, error thresholds and watermark resume.
🚀 The MCP 2026-07-28 Specification
The official 2026-07-28 release turns MCP from a bidirectional stateful protocol into a stateless request and response one. It retires the initialize handshake and Mcp-Session-Id, carrying protocol version, client identity and capabilities in per-request metadata with server/discover optional, so any request can land on any instance behind round-robin load balancing. Mcp-Method and Mcp-Name become mandatory headers so gateways and rate limiters route without parsing JSON. MRTR replaces held-open streams with input_required and inputResponses, and list results gain ttlMs and cacheScope.
Books
📚 Data Engineering for Multimodal AI
A multimodal data-engineering blueprint that ties integration-centric constructs into pipeline design. It details on-the-wire data contracts and semantic layer pipelines for trustable entity resolution, then extends to ELT/ETL tradeoffs, streaming ingestion with event-time alignment and backpressure, vector-native and hybrid storage with graph-augmented retrieval and caching, and cloud-native scaling. It closes with orchestration patterns, composite-model CI/CD, observability/runbooks, and governance/security for responsible operations.
Top comments (0)