Articles
🔍 APISIX Gateway 504s After Exactly 15 Seconds: Tracing It to tcp_syn_retries
Intermittent 504s at exactly 15 seconds on API7 Enterprise, a commercial Apache APISIX distribution, came from net.ipv4.tcp_syn_retries=3, not the 600s plugin timeout. Source tracing confirmed the timer reached the C module. Logs separated 21 DNS errors averaging 2.25s from 750 SYN timeouts averaging 15.38s, and packet capture showed retransmissions at 1s, 3s and 7s. Raising retries to 5 was rejected because it stretches failure to 63s and hides the dropped SYNs, so connectivity went to the network team. Values such as 15s, 31s and 63s mark kernel retry budgets.
🔍 Bounded Delegation for AI Agents: From Kubernetes to MCP
A Keycloak extension bounds RFC 8693 delegation for MCP calls: issued permissions are the intersection of subject token, actor token, grant and both parties' current permissions, with lifetime and hop depth inherited. Each actor token's jti authorizes one issuance through Keycloak's database-backed single-use records: across two nodes sharing PostgreSQL, 32 concurrent reuses yielded one token, while 100 distinct jti values all succeeded. The task-bound-software profile adds a parameter hash and online checks. Single use does not give exactly-once execution.
🔍 Credential Brokering Patterns for AI Agents Part 3: Protecting the Token Vault with OpenBAO
Two OpenBao designs protect agentgateway's token vault. In the first, only the gateway may call auth/jwt/login, and a token_num_uses=1 token still opens any row of that user the controller picks. In Mode 3b, a control_group on transit/decrypt holds the request. The broker reads request_data through sys/control-group/request, recomputes context from the subject token and a gateway proof, and approves one row with sys/control-group/authorize. Neither controller nor broker holds a standing decrypt. The price is four OpenBao calls per decrypt and a dependency on OpenBao 2.7.
🔍 Kafka Just Learned to Be a Queue. Postgres Has Been One All Along.
Postgres FOR UPDATE SKIP LOCKED stays the better queue when a job is born in a database transaction, because the commit is the ack, while Kafka share groups fit teams already running Kafka. On 8 vCPUs, SKIP LOCKED reached 2-2.5x the throughput of plain FOR UPDATE under 5 ms of held work. One idle REPEATABLE READ session left 200,000 dead rows unremovable and raised median claim latency from 0.27 ms to about 15 ms. Share groups bring other traps: a 30 s acquisition lock, share.auto.offset.reset=latest, batch-level sharing and Preview Python clients.
🔍 Letting a 4B Model Scale Deployments: Write Tools Behind Agent Gateway
Five independent layers let a Qwen3-4B agent call one write tool through Solo.io Agent Gateway. The kubernetes-mcp-writer instance uses enabled_tools = ["resources_scale"], a pinned image digest, and a ServiceAccount limited to deployments and deployments/scale in agent-sandbox. A new NetworkPolicy was needed because the existing selector missed the writer pods, and the mcp-admin CEL rule needed an explicit exclusion. A client-side gate treats a call as a write only when scale is present, reads current replicas, runs the frozen arguments a human approved, and writes agent-audit.jsonl.
🔍 Microsoft Copilot Studio MCP OAuth Support
Wire captures from a logging authorization server show what Microsoft Copilot Studio sends for MCP OAuth: PKCE S256 in every mode, client_secret_post with a mandatory secret in Manual mode, DCR as a public client in Dynamic discovery, and CIMD ignored. Redirect URIs are per connector and derived from the tool name. Percent-encoding case differs between authorize and token, so compare decoded values. With Shared account, the maker's token goes out for every user, and x-ms-object-id is unsigned. After a rejected refresh, Studio retries the same token each turn until a manual Reconnect.
🔍 Never let a request decide who pays
Apigee token quotas take identity, plan and size from the verified credential and gateway measurements, never from caller headers such as x-user-id or x-tier. VerifyAPIKey maps a key to its app, developer and API product, whose LLM operations carry per-model quotas. LLMTokenQuota runs twice on one SharedName: EnforceOnly rejects in the request flow, and CountOnly adds actual usage in the response flow or EventFlow. DataCapture records model, tokens and estimated cost. developer_email fills only when the proxy checks an API key or OAuth token; otherwise each report row shows (not set).
🔍 Queue or Log? Kafka, RabbitMQ and NATS All Do Both Now
Six setups on Kafka 4.3.1, RabbitMQ 4.3.6 and NATS 2.15.0 ran four failure tests. Two settings explained most results: whether storage keeps handled messages, and whether consumers ack per message or keep an offset. Since RabbitMQ 4.3, basic.nack(requeue=True) no longer counts toward x-delivery-limit, so a poison message looped 6,064 times in 45 seconds, while basic.reject dead-lettered it after 6 deliveries. A Kafka share group gave work to 3 of 6 consumers because members acquire whole producer batches, and it lost order even with max.poll.records=1.
🔍 Securing Agent-to-Agent Calls: What We Learned Putting A2A Behind a Gateway
A2A calls carry only message text, so a tool-style gateway has no skill name, user, or approval signal to enforce. Running Copilot Studio and Foundry agents through Andromeda's gateway produced four changes: govern each agent as one allow, ask, or deny action; pin the admin-approved agent card and route only to that endpoint; apply identity, policy, and credential checks to sends, polls, and cancels while refusing streaming and webhooks; and answer approval holds with a plain message any client can show. User identity across hops remains open, pending RFC 8693 or ID-JAG adoption.
🔍 Token Exchange at the API Gateway: RFC 8693 in Practice
OAuth 2.0 Token Exchange closes the hop between gateway and producer that passthrough, forwarded identity headers, and client credentials each leave open. A runnable Keycloak 26.8 realm and .NET API exchange an access token with audience api-gateway for one scoped to orders-api, keeping sub and setting azp to the gateway. Requests then fail by design: 401 on wrong audience, 401 on a bare X-User-Id header, and 403 when Bob lacks order-viewer. Vendor status covers Keycloak V2 limits, preview delegation with act, Entra's jwt-bearer on-behalf-of, and Kong support from 3.14.
AWS
🔍 Credential Brokering Patterns for AI Agents Part 2.5: Where KMS Grants Stop Scaling
AWS KMS CreateGrant defaults to 50 requests per second per account and region, and each key holds 50,000 grants, so grant-based token exchange saturates on a shared agent gateway, and stranded grants can fill a key in about 17 minutes. The revised design gives the broker standing kms:Decrypt but no ciphertext. It derives owner from the verified subject token, so KMS still fails the EncryptionContext check on a mismatched row. Because a compromised controller sees in-use plaintext, refresh tokens and OAuth client secrets move to a refresher that returns only access tokens.
🔍 One Request, Two Checkpoints: Securing AgentCore Gateway with Microsoft Entra ID
AgentCore Gateway uses CUSTOM_JWT to validate Entra tokens against the tenant's v2 discovery document, with the gateway registered as the protected API and each client registered separately. allowedClients checks client_id while Entra emits azp or appid, and delegated permissions sit in scp, so valid tokens can be rejected unless custom claim rules are added. The outbound hop uses IAM, or TOKEN_EXCHANGE for audience-specific on-behalf-of tokens instead of passthrough. Private targets connect through VPC Lattice, and 401 versus 403 shows which boundary refused a call.
Per-tenant API keys on Amazon API Gateway give each company its own token bucket, with one usage plan per tier. The CDK sample pairs the key with a Cognito JWT, and a Lambda check binds apiKeyId to the immutable custom:tenant_id claim so a key cannot consume another tenant's allowance. A traffic lab recorded 43 of 100 requests succeeding at 5 rps against a 2 rps plan while Premium stayed untouched. Design limits include 10,000 keys per account and Region, new keys inactive for 45-65 seconds, tier changes by moving keys, and Lambda concurrency returning 5xx before any 429.
🔍 Your agent has a shell. It doesn’t need MCP, it needs an identity
A cloud code agent calls internal Okta-protected APIs through a loopback proxy on 127.0.0.1 inside its AgentCore microVM, so no token reaches the model or sandbox. The harness runs RFC 8693 on-behalf-of exchange through AgentCore Identity's vault, with scopes and audience held in a server-side catalog. Testing showed Okta skips group checks on exchange, so the API reads a groups claim and returns 403. Keycloak mints only the scopes the user is entitled to. Okta and Entra emit no act claim, leaving client id plus sub as the way to recognise agent calls.
Apache Camel
🔍 Apache Camel tool calling: LLM integrations with bounded loops
Camel routes registered with the ai-tool component become LLM tools in the camel-openai chat-completion loop, with no agent framework. Bounds sit outside the prompt: tags set tool visibility, maxToolIterations caps passes, argSchema constrains arguments with JSON Schema enums, and returnDirect skips the final model round trip. camel.server.mcp-tags publishes the same routes over streamable HTTP, and the client's toolNames allow-list hides initiate_refund from one assistant while it stays live for others. The loop is stateless, so multi-step planning needs an agent runtime.
🔍 One request, several agents: semantic routing with Apache Camel and Jev
Camel routes coordinate four A2A specialist agents. A batched semantic evaluation of independent booleans (needsWeather, needsCost, needsReservation) selects specialists, a parallel Split fans out, and a Switch maps each label to a fixed a2a endpoint. A bounded Loop allows three generation attempts: relevance above 0.6 accepts, 0.4-0.6 goes to review, below 0.4 retries. Service failures return 502 through scoped onException without consuming semantic attempts. The single-specialist path skips the whole-request check, so an accepted reply does not prove every intent was found.
Apache Kafka
🔍 Connection Switching: Identity without commitment
Kroxylicious connection switching binds a virtual Kafka cluster to one physical cluster at connect time, so workload dispatch and DR cutover need no client config change: flipping a route severs connections and clients reconnect to stable endpoints. Node IDs map through downstreamId = (upstreamId * numRoutes) + routeIndex, so changing route count forces a reconnect. State stays behind: EndTxnRequest returns INVALID_TXN_STATE or UNKNOWN_PRODUCER_ID, offsets need KIP-1279 or MirrorMaker 2, and idempotent dedup does not cross the cutover. DR needs an egress proxy.
🔍 Designing Event Streams for AI Agents
AsyncAPI v3 contracts address three gaps when agents consume event streams: discovery, access and observability. A WebSocket gateway fronting Kafka maps orders/{region}/events channels to OAuth scopes and checks each subscribe, so an eu-scoped token receives scope_denied on the us channel. Cost control inverts the default: local deterministic filters first, batched model calls second, single-event reasoning last, under a dollars-per-subscription circuit breaker. Four per-channel metrics include the ratio of LLM calls to events.
🔍 Exactly-once delivery with Flink on a Kafka stream — Learnings in a lab
A Flink TaskManager was stopped mid-run under each Kafka sink: at-least-once kept all 30,000 records and wrote 701 twice, while exactly-once wrote 30,000 with no duplicates. KafkaSink pre-commits at the checkpoint barrier and commits on notifyCheckpointComplete, so after checkpoint N completes, the only copy of its records sits in the open transaction, and a broker abort loses them. transactionTimeoutMs must cover the checkpoint interval plus restart time and stay under transaction.max.timeout.ms. A cluster default kept checkpointing on when the job said off.
🔍 Exorcising Ghost Apache Kafka Controllers
Stopped KRaft controllers left RegisterControllerRecord entries that froze old feature levels and blocked Share Groups. KIP-1312's unregister-controller needs metadata.version 4.4-IV2, yet feature validation refuses that version while the ghost registrations exist, so each operation waits for the other. Aiven's pre-4.4 patch adds a `decommissioned_controller` marker to the existing record without a new metadata event; patched controllers skip marked non-voters during validation. The 4.4 forward port keeps the marker filter while adopting the native UnregisterControllerRecord.
🔍 Kafka Queues at Doctolib’s Communication Platform
Kafka Share Groups dispatch records to available consumer threads instead of assigning partitions, so Doctolib's HPA events cause no rebalance and deployments moved from Recreate to RollingUpdate. Load tests on 20 partitions raised throughput from 5,539 to about 17,500 msg/s; cutting batch size from 500 to 100 mattered because group.share.partition.max.record.locks divided by batch size caps active threads. group.share.max.size bounds HPA replicas, with a fatal GROUP_MAX_SIZE_REACHED above it. Duplicates stayed at or below 0.003% in 5 million message tests.
🔍 LinkedIn Built Kafka. What Did It Change When Kafka Was No Longer Enough?
LinkedIn's Northguard splits Kafka partition duties: ranges hold ordered keyspace logs, segments are the replication unit, and log striping lets new segments use added brokers without moving history. Metadata runs in Raft-backed vnodes (DS-RSM, 128 or more) with SWIM gossip for membership. Xinfra topic epochs let one logical topic span Kafka and Northguard, with producers dual writing during migration so rollback stays possible. All performance claims are LinkedIn's own June 2025 figures; the systems are closed source and unverified elsewhere.
🔍 Stream Branching: Mind the gap
A Kafka protocol proxy can copy produce traffic to extra clusters, but branch lines are dual writes with no undo. Fire-and-forget leaves MIA and ghost batches; dispatch on ack removes ghosts yet still duplicates on retries. Router-owned branch PIDs die with each connection, so reconnects and leadership changes create duplicates unless a state store keeps them, and transactions cannot branch at all. Per-request replica routing avoids cross-AZ fetch costs when the proxy shares the client's AZ. For backpressure the post favours drop over block, with queue depth and drop counts as metrics.
🔍 What max.poll.interval.ms bounds in Kafka Streams, measured
Kafka Streams 4.2.0 runs passes of up to N records per task between polls, so max.poll.interval.ms bounds a whole round, one call per task, not a single call. Three tasks at a 12-second call against a 30-second interval caused four evictions in 210 seconds and no commits. N has no upper limit and survives a rebalance. A fast thread reached N=222, then took 40 records at 10 seconds each as one 400-second pass and was evicted on every rejoin, with none committed. Capping max.poll.records at 20 cleared that run without eviction.
🔍 Calling an external service from Kafka Streams: size the threads, budget every call
A blocking call caps a Kafka Streams thread at one record per call. At 300 ms, one thread did 3.3 records/s, and 32 threads on 32 partitions did 106. Little's law sizes the threads, and each call gets a budget covering connect, read, retries and backoff. The JDK HttpClient's 5-second timeout did not end a slow-body response (20.05 s), so a CompletableFuture deadline wraps the call. Tasks per thread and max.poll.records, each times the budget, must stay under half of max.poll.interval.ms. By default one failed call stops the whole client, so the slow step runs as its own application.
🔍 Your Kafka Consumers Are Becoming a Workflow Engine
Kafka offsets cannot say whether a job is waiting, cancelled or safe to retry, so cancellation needs a shared decision point: a versioned transaction on the job record authorizes each step and records dispatch intent. An inbox table, an outbox and contiguous offset commits handle duplicate delivery. operation_key, built from job ID, step and input digest, stops a retry after a lost renderer response from starting a second render. Deadlines and next_attempt_at are persisted state. Temporal adds durable waits and replay, but its Activities still need idempotency.
Google Cloud
🔍 A semantic cache is easy to demo and hard to trust
Apigee's SemanticCacheLookup threshold has a direction that depends on DistanceMeasureType: 0.9 suits dot product, but on COSINE_DISTANCE nearly every prompt hits. Runtimes before 1-18-0-apigee-4 ignore the element and invert the comparison. The cache embeds only UserPromptSource, so it cannot see system prompts, history, tools or model name, and entries are shared across callers. Lookup can reach Vector Search over Private Service Connect on plaintext gRPC port 10000, while SemanticCachePopulate is REST-only through upsertDatapoints.
Kong
🔍 500 APIs, Not 500 Tools: Governing What AI Agents Can Call
The command deck file openapi2mcp turns OpenAPI operations into AI MCP Proxy tools, including POST and DELETE unless marked x-kong-mcp-exclude. Lab runs on Kong 3.14 and decK 1.68 found that only OAuth2 with x-kong-mcp-acl yields per-tool allow lists; an API-key spec converts silently into tools with no ACLs on an unauthenticated route. Domain listeners selected by tag keep tool lists small, but a consumer-group listener without include_consumer_groups gives every agent an empty list and 403. Refusals arrive as HTML, the backend sees X-Forwarded-For: unix:, and key-auth forwards the agent key.
🔍 Mirroring Without Exposing Callers: Lab-Testing an Apigee-to-Kong Cut-over
NGINX, Envoy and Istio discard a mirror's response, so this Apigee-to-Kong cut-over adds a comparator. It pairs Kong's masked answer with Apigee's by request ID, checks both against the OpenAPI contract and produces seven verdicts. In the lab with mock gateways, a slow NGINX mirror held a caller's next keep-alive request for 2.00 s, against 0.04 s with an immediate acknowledgement. On Kong 3.14, default retries sent one timed-out read to the backend six times, and mirrors rewrote Host, breaking host-based routes until X-Original-Host restored it.
🔍 Same Policies, Different Order
Apigee runs policies in written order while Kong uses fixed phases and per-phase priority, so a migration reproduces behaviour, not sequence. Conditional flows map to route regex_priority, access-phase dependencies to dynamic ordering, and custom plugin order to PRIORITY; response conflicts require moving logic to pre-function or post-function. Through Kong 3.13, dynamic ordering in a workspace silently stopped consumer-group plugins, disabling tier quotas with no error. A silver key pushed past a small quota catches it; a keyless burst expecting 401 and 429 proves spike guard order.
🔍 Token Exchange in an Apigee Exit
Kong Gateway 3.14.0.15's OpenID Connect token exchange checks only a partner token's iss, exp and nbf, so forged tokens reach WSO2 IS 7.3.0. The design puts Kong's JWT plugin (priority 1450) ahead of openid-connect (1050), with one consumer per partner issuer and keys written from Vault. WSO2 issues scopes from the local account, drops ungranted ones with a 200, and carries approved Apigee Products as a claim mapped to consumer groups. Refusals and outages all reach partners as 401 invalid_token, so alerts watch 401 rates and two log lines, not 5xx.
RabbitMQ
🔍 RabbitMQ Was Running. So Why Were Messages Stuck?
A RabbitMQ broker was OOMKilled at a 4 GiB limit although its memory alarm never fired. The watermark, 0.99 of an operator-set 80% baseline, came to 3.402 GB against RSS, while the cgroup also charged TCP socket buffers, page cache and kernel slab. SIGKILL skipped the preStop drain, a reconnection storm pushed Erlang processes from about 4,000 to 35,708, and 54,176 messages sat in 31 queues with no consumers. Lowering vm_memory_high_watermark.relative to 0.6 left 2.23 GB of headroom. An allocated_unused explanation was rejected because RSS already includes it.
🔍 When RabbitMQ Accepts a Message but Does Not Route It to a Queue
On a three-node Mnesia-backed RabbitMQ deployment, rabbitmqctl list_bindings showed topic exchange bindings that were incomplete in the runtime tables rabbit_topic_trie_edge and rabbit_topic_trie_binding, so publishes returned 'Message published, but not routed'. A utility reconstructs binding paths from ets:tab2list output and compares them with configured topology after restarts, rescheduling and upgrades. Exchange-level unrouted metrics narrow the fault. Deleting and recreating the binding through the API was the deployment-specific workaround.
SAP
🔍 End-to-End Principal Propagation for AI Agents with SAP Integration Suite, MCP and SAP Cloud Connect
SAP Integration Suite's MCP Server exposes a Node.js backend through an OpenAPI 3.0.3 contract. OAuth authorization_code against XSUAA supplies the human principal, and a BTP destination uses ProxyType=OnPremise with Authentication=PrincipalPropagation. Cloud Connector authenticates over mTLS and forwards a short-lived X.509 user certificate as SSL_CLIENT_CERT. The backend verifies the TLS peer before parsing that header, then applies its own RBAC: a SALES user saw 0 of 4 orders. Documented failures include missing EKU and unresolved ${mail}.
Releases
Apache Camel 4.23 resolves 1030 JIRA tickets, about 145 of them security hardening. New Switch and Cache EIPs join a KeyValueRepository SPI with 11 backends, used by the Idempotent Consumer, Aggregator and state-store component. The Preview semantic language routes on message meaning. camel-kafka-share consumes KIP-932 share groups with per-record ACCEPT, RELEASE and REJECT. New OPA, OpenFGA and SPIFFE components fail closed, and camel-ai-observability records OpenTelemetry GenAI spans. Groovy routes run about 4.3x faster, and TLSv1.2 becomes the default minimum.
Apache Kafka 4.4.0 ships 25 KIPs. KIP-1191 adds dead-letter queues for share groups once share.version reaches 2. KIP-1312 adds controller unregistration through kafka-cluster.sh unregister-controller, requiring metadata version 4.4-IV2. KIP-892 adds transactional state stores behind enable.transactional.statestores, so exactly-once errors replay only uncommitted writes. KIP-1357 adds broker-side custom task assignors for Streams groups, KIP-909 adds asynchronous bootstrap DNS resolution, and KIP-1209 lets Connect refuse to auto-create internal topics.
Pulsar 5.0 LTS adds Scalable Topics, addressed with the topic:// scheme, where key-range segments split and merge at runtime while stream consumers preserve per-key ordering. The v5 Java client replaces subscription types with stream, queue and checkpoint consumers, and Oxia is the recommended metadata store for new clusters. Existing topics and the v4 client stay supported. In a 500-publisher Key_Shared benchmark, p99 latency fell from 202.3 s on 4.0.13 to 1.08 s. Servers require Java 21, and Scalable Topics lack geo-replication.
RabbitMQ 4.4 adds producer-set delayed messages through x-opt-delivery-time and consumer-deferred messages on quorum queues. Consumers defer with x-opt-deferral-token and retrieve through rabbitmq:deferral-tokens, which implements the Resequencer pattern without an external store. Members can spread across failure domains using quorum_queue.member_placement_tag. Native STOMP, AMQP 1.0 connection uniqueness and Erlang/OTP 29 post-quantum TLS also ship. Tanzu RabbitMQ 4.4 adds leader migration, JMS topics over AMQP 1.0, and exchange federation v2 with 2 connections regardless of exchange count.
Top comments (0)