DEV Community: SoftwareDevs mvpfactory.io

Pruning and Quantizing CoreML Models for Real-Time On-Device Inference: Cutting Latency in Half Without Accuracy Loss

SoftwareDevs mvpfactory.io — Fri, 17 Jul 2026 14:49:55 +0000

---
title: "Pruning and Quantizing CoreML Models for Real-Time On-Device Inference"
published: true
description: "Walk through Apple's coremltools palettization and unstructured pruning APIs to cut on-device latency by 50%+ on Neural Engine targets without sacrificing model accuracy."
tags: ios, swift, mobile, architecture
canonical_url: https://mvpfactory.co/blog/coreml-model-compression-latency
---

## What You Will Build

By the end of this walkthrough, you will have a CoreML compression pipeline that combines INT4 palettization and unstructured pruning to cut inference latency by 50–65% on Neural Engine targets — without blindly nuking your model's accuracy. We will cover the profiling workflow, the quantization order that preserves calibration integrity, and the pruning config that compounds your gains.

A typical MobileNet-class float32 model at ~14MB runs at roughly 8ms on the A17 Neural Engine. INT8 quantization drops that to ~5ms. Selective INT4 palettization gets you to ~3.5ms. That is a 56% latency reduction from a single compression pass, and most teams are not doing it.

---

## Prerequisites

- Python 3.9+ with `coremltools 7.x` installed
- A trained PyTorch model (MobileNet-class or similar)
- A calibration dataloader (128+ batches recommended)
- Xcode 15+ to validate the output `.mlpackage`

---

## Step 1: Profile Before You Compress

Let me show you a pattern I use in every project. Never compress blindly. Start with per-layer sensitivity analysis using `get_weights_metadata`:

python
import numpy as np
import coremltools as ct
from coremltools.optimize.coreml import get_weights_metadata

model = ct.models.MLModel("MyModel.mlpackage")
metadata = get_weights_metadata(model, weight_threshold=1024)

for name, info in metadata.items():
w = info.val
sparsity = np.mean(np.abs(w) < 1e-6)
print(f"{name}: shape={w.shape}, sparsity={sparsity:.2%}, dtype={w.dtype}")


Computing near-zero sparsity from `info.val` gives you ground truth on which layers already trend sparse. Convolutional layers in early blocks typically tolerate 70–80% induced sparsity with less than 0.5% accuracy drop. Attention layers and final classification heads are where you pay the real cost — keep those at float16 or conservative INT8.

---

## Step 2: Know Your Quantization Options

| Quantization | Size Reduction | Latency Gain (Neural Engine) | Accuracy Risk |
|---|---|---|---|
| Float16 (baseline) | 2x vs FP32 | — | Negligible |
| INT8 (linear) | 4x vs FP32 | ~35–40% | Low if calibrated |
| INT4 (palettized) | 8x vs FP32 | ~50–60% | Medium — layer-dependent |
| Mixed INT4/INT8 | 5–6x vs FP32 | ~45–55% | Low with profiling |

INT4 palettization benefits Neural Engine execution disproportionately because the ANE's memory bandwidth bottleneck is the primary constraint, not compute. On GPU execution units, INT4 gains shrink and you risk latency regressions from dequantization overhead. Mixed precision is almost always the right production choice.

---

## Step 3: The Conversion Pipeline That Does Not Break Calibration

Here is the gotcha that will save you hours. If you run post-training quantization on the ONNX graph *before* CoreML conversion, you lose the operator-level visibility coremltools needs to set accurate activation ranges per layer. Quantize in PyTorch space first, then trace.

python
import torch
import coremltools as ct
from coremltools.optimize.torch.quantization import PostTrainingQuantizer, PostTrainingQuantizerConfig

Step 1: Quantize in PyTorch space BEFORE tracing

config = PostTrainingQuantizerConfig.from_dict({
"global_config": {
"weight_dtype": "int8",
"activation_dtype": "int8"
}
})
quantizer = PostTrainingQuantizer(model, config)
quantized_model = quantizer.compress(dataloader=calibration_loader, num_batches=128)

Step 2: Trace the quantized model

example_input = torch.rand(1, 3, 224, 224)
traced = torch.jit.trace(quantized_model, example_input)

Step 3: Convert — activation ranges are already embedded

mlmodel = ct.convert(traced, inputs=[ct.ImageType(shape=example_input.shape)])
mlmodel.save("CompressedModel.mlpackage")


128 calibration batches is the practical floor. Below 64, activation range estimates drift enough to cause silent accuracy degradation that only surfaces on edge-case inputs — exactly the kind of bug that clears your eval suite and then blows up in production.

---

## Step 4: Add Unstructured Pruning for Compounded Gains

The docs do not emphasize this enough, but pruning and quantization stack. Here is the minimal setup to get this working:

python
from coremltools.optimize.coreml import OpThresholdPrunerConfig, prune_weights

pruner_config = OpThresholdPrunerConfig(
threshold=1e-3,
minimum_sparsity_percentile=0.4,
maximum_sparsity_percentile=0.8,
weight_threshold=1024
)
pruned_model = prune_weights(model, config=pruner_config)


Unstructured pruning at 50–70% sparsity on feed-forward layers compounds with INT8 quantization — you get both memory bandwidth and compute wins. INT4 palettization alone gets you ~50% latency reduction. Adding 60% unstructured sparsity on compatible layers pushes that to 60–65%. That is the difference between a demo and something shippable.

(Speaking of apps that need to stay performant in the background — [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) is a good reminder that even utility apps benefit from lean on-device inference when they are competing with your IDE for CPU budget.)

---

## Gotchas

**Compressing attention heads will hurt you.** Sensitivity analysis exists for a reason. Blindly applying INT4 to attention layers is how you ship a model that passes eval and breaks in production on real-world input distributions.

**Post-ONNX PTQ fails silently.** This is the most common pipeline mistake I see. The failure mode does not surface on standard benchmarks — it appears on tail inputs, usually after launch.

**GPU vs Neural Engine targets differ.** INT4 gains shrink significantly on GPU execution units. If your deployment target could be either, run separate profiling passes and use mixed precision accordingly.

**128 calibration batches is a floor, not a suggestion.** Under 64 batches, your activation ranges become unreliable. Use your real training distribution, not synthetic data.

---

## Conclusion

Profile per-layer with `get_weights_metadata`, quantize in PyTorch space before tracing, and combine INT4/INT8 mixed precision with 60% unstructured sparsity on feed-forward layers. That pipeline consistently delivers 60–65% latency reduction on Neural Engine targets with manageable accuracy trade-offs. Skip sensitivity analysis and you are guessing — and on-device AI ships or dies on milliseconds.

**Relevant docs:** [coremltools optimization guide](https://apple.github.io/coremltools/docs-guides/source/optimization-workflow.html) · [Neural Engine performance best practices](https://developer.apple.com/documentation/coreml)

Batched Prompt Scheduling for On-Device LLMs

SoftwareDevs mvpfactory.io — Fri, 17 Jul 2026 08:07:58 +0000

---
title: "Batched Prompt Scheduling for On-Device LLMs: Priority Queues, Preemption, and the Android Inference Engine That Never Stalls the UI"
published: true
description: "Build a three-layer Android inference scheduler that multiplexes on-device LLM calls across foreground chat, inline suggestions, and background summarization — with preemption, partial KV cache eviction, and a lifecycle-aware token-budget governor."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/android-on-device-llm-scheduling
---

What We Are Building

By the end of this tutorial you will have a working three-layer inference scheduler that multiplexes on-device LLM calls across concurrent Android callers — foreground chat, inline suggestions, and background summarization — without canceling in-flight jobs or stalling the UI.

Here is the problem we are solving first: on a Tensor G3 chip, a 3B parameter model saturates roughly 85% of the NPU during active generation. A foreground chat request arriving mid-inference on a background summary job sees first-token latency jump from 140ms to over 1,400ms. That is a 10× regression from a single design oversight.

Prerequisites

Android project targeting API 26+
A running on-device inference engine (MediaPipe, llama.cpp JNI, or similar)
androidx.lifecycle:lifecycle-process dependency for ProcessLifecycleOwner

The Three-Layer Architecture

┌─────────────────────────────────────────┐
│           InferenceOrchestrator          │
│  ┌──────────────┐  ┌───────────────┐    │
│  │ PriorityQueue│  │ TokenBudget   │    │
│  │ (min-heap)   │  │ Governor      │    │
│  └──────────────┘  └───────────────┘    │
│         │                  │             │
│  ┌──────▼──────────────────▼──────────┐ │
│  │        PreemptionController        │ │
│  │  (KV cache snapshot + eviction)    │ │
│  └────────────────────────────────────┘ │
│                    │                     │
│         ┌──────────▼──────────┐          │
│         │   InferenceEngine   │          │
│         └─────────────────────┘          │
└─────────────────────────────────────────┘

Step 1 — Priority Queue With Preemption

Assign each request a RequestPriority based on caller type:

Caller Type	Priority	Max Token Budget	Preemptible
Foreground Chat	CRITICAL (0)	Unlimited	No
Inline Suggestion	HIGH (1)	128 tokens	No
Background Summary	NORMAL (2)	512 tokens	Yes
Offline Indexing	LOW (3)	1024 tokens	Yes

The queue is a PriorityBlockingQueue<InferenceJob> backed by a min-heap on priority ordinal. Let me show you a pattern I use in every project — the PreemptionController:

class PreemptionController(private val engine: InferenceEngine) {

    fun preempt(current: InferenceJob, incoming: InferenceJob) {
        if (incoming.priority < current.priority && current.isPreemptible) {
            val snapshot = engine.snapshotKVCache(current.sessionId)
            current.checkpointState = snapshot
            engine.suspendGeneration(current.sessionId)
            engine.resume(incoming.sessionId)
        }
    }
}

Notice what is not here: a cancellation call. You snapshot the KV cache and park the job. When the high-priority request completes, the background job resumes from that checkpoint.

Step 2 — Partial KV Cache Eviction

Full eviction forces re-prefilling the entire prompt context on resume — expensive. The minimal setup that works: retain the static system-prompt portion of the KV cache (it never changes) and evict only the dynamic conversation turns.

Retaining the first N layers of the KV cache during preemption cuts re-prefill cost by 40–60% on prompts with fixed system instructions exceeding 512 tokens. Cache eviction only fires when memory pressure exceeds 70% of available LPDDR5 bandwidth.

Step 3 — Token-Budget Governor Tied to Process Lifecycle

Here is the gotcha that will save you hours. Most implementations wire the budget governor to a timer. Timers are guesses. Android's ProcessLifecycleOwner exposes authoritative ON_RESUME and ON_STOP events — use those instead:

class TokenBudgetGovernor(lifecycle: ProcessLifecycle) {

    private var budgetMultiplier = 1.0f

    init {
        lifecycle.addObserver { event ->
            budgetMultiplier = when (event) {
                ON_RESUME  -> 1.0f   // full budget, foreground
                ON_STOP    -> 0.25f  // aggressive throttle, background
                else       -> 0.5f
            }
        }
    }

    fun budgetFor(job: InferenceJob): Int =
        (job.priority.baseBudget * budgetMultiplier).toInt()
}

When the app moves to background, token output throttles to 25% of base budget. Battery savings are a side effect — the real goal is ensuring the next foreground request hits the engine with full resources already reclaimed.

Latency Results

Scenario	Naive Queue	Priority Scheduler
Foreground chat, no background jobs	138ms TTFT	135ms TTFT
Foreground chat, background summary active	1,420ms TTFT	161ms TTFT
Inline suggestion under load	890ms TTFT	148ms TTFT
Background resume after preemption	N/A (canceled)	+22% vs fresh start

The 22% overhead on background job resume is the cost of partial KV cache re-prefill. That is the right trade.

Gotchas

Do not model this like a coroutine dispatcher. Throwing requests at a shared dispatcher and hoping for the best works with one caller. With three concurrent callers it produces that 1,400ms spike shown above.

Four priority tiers is the right number. CRITICAL, HIGH, NORMAL, LOW covers the full production caller space. More granularity adds scheduling overhead with negligible SLA benefit.

Lifecycle events, not timers. The docs do not make this obvious, but ProcessLifecycleOwner is the correct authority for budget decisions. A Handler-based poll introduces 100–500ms of lag at exactly the moment precision matters.

Conclusion

A three-layer scheduler — priority queue with preemption, partial KV cache eviction, and a lifecycle-aware token budget governor — brings foreground chat from 1,420ms down to 161ms TTFT under load without canceling a single background job.

Relevant reading: Android ProcessLifecycleOwner docs, MediaPipe LLM Inference API.

WebSocket Multiplexing Over HTTP/2: Building a Single-Connection Real-Time Layer for Mobile APIs Without SSE Complexity

SoftwareDevs mvpfactory.io — Thu, 16 Jul 2026 13:18:34 +0000

---
title: "WebSocket Multiplexing Over HTTP/2: The Mobile Real-Time Fix You Did Not Know You Needed"
published: true
description: "HTTP/2 silently breaks WebSocket upgrades on mobile. Here is the connect-protocol + Traefik configuration that gives you true bidirectional streaming over a single persistent connection."
tags: mobile, api, architecture, performance
canonical_url: https://blog.mvpfactory.co/websocket-http2-mobile-real-time
---

What We Will Build

By the end of this tutorial you will understand why your WebSocket connections are almost certainly falling back to HTTP/1.1 without any warning, and you will have working OkHttp (Kotlin) and URLSession (Swift) configurations using connect-protocol to get true bidirectional streaming over a single HTTP/2 connection — wired through a correctly configured Traefik reverse proxy.

Prerequisites

Android project using OkHttp, or iOS project targeting iOS 15+
Traefik as your reverse proxy
A backend service that speaks h2c or h2
Familiarity with gRPC-style streaming APIs

The Problem You Probably Have Right Now

Let me show you a pattern I see in nearly every mobile codebase I audit.

The team instruments their app, sees "WebSocket connected," and assumes the connection is riding their HTTP/2 infrastructure. It is not.

WebSocket depends on the Upgrade: websocket header — a purely HTTP/1.1 construct. HTTP/2 has no upgrade mechanism. It uses binary framing and stream multiplexing from the first byte. RFC 8441 introduced an extended CONNECT method to tunnel WebSocket over HTTP/2, but support across clients, proxies, and load balancers remains inconsistent.

In practice, when OkHttp or URLSession attempt a WebSocket connection against an HTTP/2 server, TLS ALPN negotiation either falls back to http/1.1 or the connection is rejected. You get one TCP connection per WebSocket — defeating the entire point of HTTP/2.

Feature	HTTP/1.1 WebSocket	HTTP/2 Stream
Upgrade mechanism	`Upgrade: websocket` header	Not supported natively
Multiplexing	No — one conn per socket	Yes — 100 concurrent streams
Proxy compatibility	Excellent	Poor without RFC 8441

TCP connection overhead on LTE runs 200–400ms (Grigorik, High Performance Browser Networking, O'Reilly). Three concurrent WebSocket channels means three connections and three TLS handshakes. With HTTP/2 multiplexing, those three streams share one.

Step 1 — Android/Kotlin: OkHttp + connect-kotlin

Here is the minimal setup to get this working. Use the connect-kotlin library from Buf:

val okHttpClient = OkHttpClient.Builder()
    .protocols(listOf(Protocol.HTTP_2, Protocol.HTTP_1_1))
    .connectTimeout(10, TimeUnit.SECONDS)
    .build()

val protocolClient = ProtocolClient(
    httpClient = ConnectOkHttpClient(okHttpClient),
    config = ProtocolClientConfig(
        host = "https://api.example.com",
        networkProtocol = NetworkProtocol.CONNECT,
        codec = ProtoCodec()
    )
)

val stub = ChatServiceClient(protocolClient)

val stream = stub.chat(headers = emptyMap())
stream.sendMessage(ChatRequest(text = "hello"))

Wrap OkHttpClient in ConnectOkHttpClient and specify NetworkProtocol.CONNECT. Force HTTP_2 first in the protocols list — OkHttp will not fall back unless the server explicitly negotiates http/1.1 via ALPN.

Step 2 — iOS/Swift: URLSession + connect-swift

let configuration = URLSessionConfiguration.default
configuration.httpAdditionalHeaders = [
    "Content-Type": "application/connect+proto"
]

let client = ProtocolClient(
    httpClient: URLSessionHTTPClient(configuration: configuration),
    config: ProtocolClientConfig(
        host: "https://api.example.com",
        networkProtocol: .connect,
        codec: ProtoCodec()
    )
)

URLSession on iOS 15+ supports HTTP/2 natively. The connect-swift library handles framing — no custom stream management required.

Step 3 — Traefik Configuration

The docs do not make this obvious, but two settings here are non-negotiable:

entryPoints:
  websecure:
    address: ":443"
    http:
      tls: {}

serversTransport:
  myTransport:
    forwardingTimeouts:
      responseHeaderTimeout: "0s"

http:
  routers:
    api:
      rule: "Host(`api.example.com`)"
      service: backend
      tls: {}
  services:
    backend:
      loadBalancer:
        servers:
          - url: "h2c://backend:8080"
        serversTransport: myTransport

The h2c:// scheme routes to your backend over HTTP/2 cleartext. The responseHeaderTimeout: "0s" disables the default timeout that silently kills long-lived streams.

Gotchas

responseHeaderTimeout is not optional. Traefik will terminate long-lived streams after its default timeout. Set it to "0s" before you go to production — this is the one that bites people most often.

Verify ALPN before you ship anything. Use Charles Proxy or Proxyman to confirm which protocol your connections actually negotiate. If you see http/1.1 in ALPN, you are leaving performance on the table.

RFC 8441 is not a reliable fix. Support across clients, proxies, and load balancers is still inconsistent. connect-protocol sidesteps this entirely by never using the WebSocket upgrade mechanism.

Conclusion

The WebSocket-over-HTTP/1.1 fallback is one of the most common invisible performance regressions on mobile. connect-protocol, combined with correct Traefik configuration, eliminates the problem at the transport layer and gives you type-safe generated clients for both Kotlin and Swift via Buf's libraries.

Verify your ALPN negotiation today — you may be surprised what you find.

Subscription Paywall Architecture for KMP Apps

SoftwareDevs mvpfactory.io — Thu, 16 Jul 2026 08:19:14 +0000

---
title: "KMP Subscription Architecture: One Engine, Two Platforms"
published: true
description: "Build a shared KMP subscription engine with expect/actual interfaces, server-side receipt validation, and a single entitlement state machine for iOS and Android."
tags: kotlin, mobile, architecture, android
canonical_url: https://mvpfactory.co/blog/kmp-subscription-architecture
---

## What We Are Building

By the end of this tutorial you will have a shared Kotlin Multiplatform subscription engine that abstracts RevenueCat, StoreKit 2, and Google Play Billing behind a single `expect`/`actual` interface, runs one entitlement state machine for both platforms, and handles grace periods, billing retries, and cache invalidation from a single source of truth.

No more duplicating billing edge-case logic across Swift and Kotlin.

## Prerequisites

- Kotlin Multiplatform project with `commonMain`, `iosMain`, and `androidMain` source sets
- RevenueCat SDK configured on both platforms (or your preferred billing provider)
- A backend endpoint for server-side receipt validation
- Familiarity with Kotlin coroutines and sealed classes

---

## Step 1 — Define the Purchase Interface in commonMain

Let me show you a pattern I use in every project. Define the abstraction first, before you touch any platform SDK.

kotlin
// commonMain
interface PurchaseProvider {
suspend fun purchase(productId: String): PurchaseResult
suspend fun restorePurchases(): List
suspend fun getOfferings(): List
}

expect fun createPurchaseProvider(): PurchaseProvider


Each platform delivers its own `actual` implementation:

kotlin
// iosMain
actual fun createPurchaseProvider(): PurchaseProvider = RevenueCatIOSProvider()

// androidMain
actual fun createPurchaseProvider(): PurchaseProvider = RevenueCatAndroidProvider()


The `expect`/`actual` boundary keeps your shared business logic decoupled from any specific vendor — including RevenueCat itself.

---

## Step 2 — Normalize the Entitlement Contract

iOS validates via the App Store Server API with JWT-signed JWS transactions. Android uses Google Real-Time Developer Notifications via Pub/Sub. Two completely different mechanisms — but your shared module should not care about either.

Your backend exposes a single `/validate-receipt` endpoint and returns a platform-agnostic payload:

kotlin
// commonMain
data class EntitlementState(
val isActive: Boolean,
val expiresAt: Instant,
val inGracePeriod: Boolean,
val billingRetryActive: Boolean,
val billingRetryUntil: Instant?
)


Here is what your backend absorbs so your KMP module does not have to:

| Concern | iOS | Android |
|---|---|---|
| Validation method | JWS transaction (JWT) | Purchase token via REST |
| Real-time events | App Store Server Notifications | RTDN via Cloud Pub/Sub |
| Grace period signal | `expirationIntent` field | `paymentState = 0` |
| Billing retry window | Up to 60 days | Up to 30 days |

---

## Step 3 — Build the State Machine in Shared Code

This is where most teams lose real money. Grace periods and billing retries gate feature access — model them as first-class states, not UI edge cases.

kotlin
sealed class SubscriptionState {
object Active : SubscriptionState()
data class GracePeriod(val expiresAt: Instant) : SubscriptionState()
data class BillingRetry(val retryUntil: Instant) : SubscriptionState()
object Expired : SubscriptionState()
object NeverSubscribed : SubscriptionState()
}

class SubscriptionStateMachine(
private val provider: PurchaseProvider,
private val cache: EntitlementCache
) {
fun resolve(entitlement: EntitlementState): SubscriptionState = when {
entitlement.isActive -> SubscriptionState.Active
entitlement.inGracePeriod -> SubscriptionState.GracePeriod(entitlement.expiresAt)
entitlement.billingRetryActive && entitlement.billingRetryUntil != null ->
SubscriptionState.BillingRetry(entitlement.billingRetryUntil)
entitlement.expiresAt < Clock.System.now() -> SubscriptionState.Expired
else -> SubscriptionState.NeverSubscribed
}

suspend fun refresh(): EntitlementState {
    val fresh = provider.restorePurchases()
    val state = resolveFromResults(fresh)
    cache.write(state)
    return state
}

}


This runs entirely in `commonMain`. Your SwiftUI views and Compose screens observe the same `SubscriptionState` flow — no platform-specific branching required.

---

## Step 4 — Cache Without Race Conditions

Cache on device, validate server-side asynchronously, and never block the UI on cold start. Here is the minimal setup to get this working: a two-layer strategy using local `EncryptedSharedPreferences`/Keychain for instant reads, with a background refresh triggered on app foreground. Invalidate eagerly on any purchase event.

Five minutes is a reasonable TTL default — short enough that a cancelled subscriber won't keep seeing premium content, long enough to avoid hammering your validation endpoint on typical mobile sessions. Make it configurable so you can tighten it for high-value paywalls without shipping a release.

---

## Gotchas

**Starting with the platform SDK instead of the interface.** The docs do not mention this, but the abstraction boundary also gives you a clean seam for unit testing your entire subscription lifecycle without a physical device or a live App Store sandbox account. Define `PurchaseProvider` in `commonMain` first, always.

**Treating grace periods as UI state.** A subscriber in a grace period still has active access — but your state machine needs to know when to show a recovery paywall. If `inGracePeriod` and `billingRetryActive` are not first-class states, you will ship a bug on one platform that was already fixed on the other six months earlier.

**Putting receipt validation logic in the client.** JWT handling for the App Store Server API and RTDN for Google Play belong on your backend. Return a normalized `EntitlementState` with a dedicated `billingRetryUntil` field and let shared code consume it cleanly.

**Hardcoding the cache TTL.** Five minutes works for most paywalls. For high-value content you will want to tighten this without a release cycle. Make it configurable from day one.

---

## Conclusion

Six subscription concerns across two platforms used to mean twelve code paths to maintain. With KMP's `expect`/`actual` pattern, a normalized backend contract, and a state machine living entirely in `commonMain`, you collapse that to one engine with thin platform adapters. The billing edge case you fix today gets fixed on both platforms simultaneously.

**Resources:**
- [Kotlin Multiplatform — expect/actual](https://kotlinlang.org/docs/multiplatform-expect-actual.html)
- [App Store Server API](https://developer.apple.com/documentation/appstoreserverapi)
- [Google Play Developer API — Subscriptions](https://developer.android.com/google/play/billing/getting-ready)
- [RevenueCat Kotlin Multiplatform](https://www.revenuecat.com/docs/getting-started/installation/kotlin-multiplatform)

PostgreSQL Index-Only Scans and Visibility Maps: The Query Optimization Layer Most Developers Never Reach

SoftwareDevs mvpfactory.io — Wed, 15 Jul 2026 13:40:05 +0000

---
title: "PostgreSQL Index-Only Scans: The Hidden VACUUM Dependency Your Covering Indexes Depend On"
published: true
description: "Why your PostgreSQL covering indexes silently fall back to heap access — and how to tune VACUUM and visibility maps for high-write SaaS workloads."
tags: postgresql, performance, architecture, devops
canonical_url: https://mvpfactory.co/blog/postgresql-index-only-scans-visibility-map
---

## What we will cover

Today you will learn why your covering index is lying to you. `EXPLAIN` says `Index Only Scan`. The plan looks clean. But heap fetches are still happening — silently, expensively — and your query performance is paying the price. We are going to diagnose this using real query plans, fix it with targeted autovacuum tuning, and build a weekly monitoring query you can drop straight into production.

(Side note: if you are the kind of developer who goes deep on database internals, you probably spend long hours at your desk. I have been using [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) for break reminders and guided desk exercises — worth keeping in your toolkit alongside your SQL profiling habits.)

## Prerequisites

- PostgreSQL 13+ (examples run on PG 15)
- Access to `pg_stat_user_tables` and `pg_stat_progress_vacuum`
- A table with meaningful write volume — event logs and audit tables are the most common offenders

---

## Step 1 — Understand what index-only scans actually require

Here is the pattern I explain in every PostgreSQL review I do. Developers learn requirement one: all columns in the query must exist in the index. Create a covering index, ship it, done.

They miss requirement two — and this is the one that kills you in production.

PostgreSQL must confirm tuple visibility **without touching the heap**. It does this via the **visibility map** — a compact bitmap, one bit per heap page. When the all-visible bit is set, VACUUM has confirmed every tuple on that page is visible to all transactions. Index-only scan proceeds cleanly. When that bit is not set — because writes have dirtied the page since the last VACUUM — PostgreSQL falls back to a heap fetch. Your "optimized" query just became a regular index scan with extra steps.

## Step 2 — Read the signals in EXPLAIN and pg_stat_user_tables

Run this on your target query:

sql
EXPLAIN (ANALYZE, BUFFERS)
SELECT user_id, created_at FROM events
WHERE tenant_id = 42 AND created_at > now() - interval '7 days';


Watch for this in the output:

sql
Index Only Scan using idx_events_covering on events
Heap Fetches: 18402
Buffers: shared hit=4821 read=3107


Non-zero `Heap Fetches` is the smoking gun. Now cross-reference with `pg_stat_user_tables`:

sql
SELECT
relname,
n_live_tup,
n_dead_tup,
round(100.0 * n_dead_tup / nullif(n_live_tup + n_dead_tup, 0), 2) AS dead_ratio_pct,
last_autovacuum,
last_vacuum,
n_mod_since_analyze
FROM pg_stat_user_tables
WHERE relname = 'events'
ORDER BY n_dead_tup DESC;


If `dead_ratio_pct` is above 5% and `last_autovacuum` is hours or days old on an active table, you have found your problem.

## Step 3 — Tune autovacuum for high-write SaaS workloads

The docs do not make this obvious, but autovacuum's defaults were designed for moderate OLTP workloads — not a SaaS backend processing thousands of events per minute. The default `autovacuum_vacuum_scale_factor` of 0.20 means autovacuum will not trigger on a 10-million-row table until 2 million rows have been modified. That is a long window to accumulate dead tuples.

Here is the minimal setup to get this working for high-write tables:

| Parameter | Default | High-Write Recommendation |
|---|---|---|
| `autovacuum_vacuum_scale_factor` | 0.20 | 0.01–0.05 |
| `autovacuum_vacuum_threshold` | 50 rows | 100–500 rows |
| `autovacuum_vacuum_cost_delay` | 2ms | 0–1ms |
| `autovacuum_max_workers` | 3 | 5–8 |
| `autovacuum_naptime` | 1 min | 15–30 sec |

Apply this per-table rather than globally — more surgical, fewer surprises:

sql
ALTER TABLE events SET (
autovacuum_vacuum_scale_factor = 0.02,
autovacuum_vacuum_cost_delay = 1
);


## Step 4 — Prime the visibility map and verify

Once autovacuum is tuned, run a manual VACUUM to recover existing tables immediately:

sql
VACUUM (ANALYZE, VERBOSE) events;


Monitor large tables with `pg_stat_progress_vacuum`. Then re-run your `EXPLAIN (ANALYZE, BUFFERS)`. On a 50-million-row events table ingesting roughly 8,000 writes per minute on PostgreSQL 15 (c5.2xlarge, gp3 storage), this dropped heap fetches from ~20,000 per query to under 50, with query latency falling by approximately 60%.

Let me show you a pattern I use in every production database I maintain — a weekly diagnostic query to stay ahead of this:

sql
SELECT
relname,
pg_size_pretty(pg_total_relation_size(relid)) AS total_size,
round(100.0 * n_dead_tup / nullif(n_live_tup + n_dead_tup, 0), 2) AS dead_pct,
age(relfrozenxid) AS xid_age,
last_autovacuum
FROM pg_stat_user_tables
JOIN pg_class ON pg_stat_user_tables.relid = pg_class.oid
WHERE schemaname = 'public'
AND n_live_tup > 10000
ORDER BY dead_pct DESC NULLS LAST
LIMIT 20;


---

## Gotchas

**`EXPLAIN` without `ANALYZE` will not show heap fetches.** The node name `Index Only Scan` appears in the plan regardless of whether heap fetches are happening at runtime. Always use `EXPLAIN (ANALYZE, BUFFERS)` to get actual execution data.

**Global autovacuum changes affect every table.** Tune per-table with `ALTER TABLE ... SET (...)` on your hottest tables first. Avoid changing cluster-wide settings without understanding your full workload mix.

**A freshly created covering index is not enough.** Existing heap pages will not have their all-visible bits set until VACUUM runs. After adding a covering index to a large, active table, run `VACUUM ANALYZE` manually before expecting index-only scan to deliver its full benefit.

---

## Conclusion

Covering indexes are only half the story. The visibility map is the invisible layer between your index strategy and actual query performance. Verify with `EXPLAIN (ANALYZE, BUFFERS)`, tune autovacuum per-table with a scale factor of 1–5% on write-heavy workloads, and treat VACUUM as a first-class performance concern rather than a maintenance afterthought. Your event logs and audit tables will thank you.

**Resources:**
- [PostgreSQL VACUUM documentation](https://www.postgresql.org/docs/current/sql-vacuum.html)
- [pg_stat_user_tables reference](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ALL-TABLES-VIEW)
- [Autovacuum tuning guide (PostgreSQL wiki)](https://wiki.postgresql.org/wiki/Autovacuum)

gRPC Bidirectional Streaming with Backpressure: Building a Real-Time Mobile API That Doesn't Collapse Under Load

SoftwareDevs mvpfactory.io — Wed, 15 Jul 2026 07:34:19 +0000

---
title: "gRPC Bidirectional Streaming on Android: Backpressure Done Right at 10k Streams"
published: true
description: "Flow control windows, cancellation propagation, and the OkHttp/gRPC-Kotlin coroutine bridge that keeps your streaming endpoint alive under real load — including Netty tuning for 10k concurrent streams."
tags: kotlin, android, mobile, api
canonical_url: https://blog.mvpfactory.co/grpc-bidirectional-streaming-android-backpressure
---

## What We Are Building

By the end of this tutorial you will have a production-ready gRPC bidirectional streaming setup on Android. Specifically: a correctly wired OkHttp/gRPC-Kotlin coroutine bridge with explicit HTTP/2 flow control windows, structured cancellation that sends `RST_STREAM` automatically, and a Netty server tuned to handle 10,000 concurrent streams without an OOM kill.

This is not a hello-world streaming example. This is the setup I use in every project where streaming matters.

## Prerequisites

- Android project targeting API 24+
- Kotlin coroutines (`kotlinx-coroutines-android`)
- `grpc-kotlin-stub` and `grpc-okhttp` on the client
- `grpc-netty` on the server side
- Basic familiarity with gRPC service definitions and protobuf

---

## Why This Is Harder Than It Looks

At 1,000 concurrent streams your app feels fine. At 10,000 it OOM-kills — and the default Netty configuration is usually why.

Bidirectional streaming removes the natural backpressure that REST and unary gRPC provide. With those patterns, the client controls when it makes the next request. Bidirectional streaming removes that constraint entirely. Your server can now push frames faster than the Android client can process them, and when it does, you get buffer bloat, OOM kills, and head-of-line stalls where one slow stream degrades every stream on the same HTTP/2 connection.

Here is the math that makes this a real problem. At 10,000 concurrent streams on a single Netty server, the default `INITIAL_WINDOW_SIZE` of 65,535 bytes per stream means roughly **625 MB of flow control buffer space** before sending a single byte of application payload:

10,000 streams × 65,535 bytes ÷ 1,048,576 = ~625 MB


That is not a theoretical concern. It is a runtime memory cliff.

---

## Step 1: Tune Your HTTP/2 Flow Control Windows

HTTP/2 defines two levels of flow control: connection-level and stream-level. Both must be tuned independently. Here is the pattern I follow:

| Parameter | Default | Recommended for Mobile | Why |
|---|---|---|---|
| Stream initial window | 65,535 bytes | 256 KB–1 MB | 512 KB is a safe starting point for most mobile payloads |
| Connection window | 65,535 bytes | 4–8 MB | Must exceed per-stream window so the connection is never the bottleneck |
| Max concurrent streams | Unlimited | 100–250 per connection | Caps memory exposure and keeps latency predictable |
| Max frame size | 16,384 bytes | 16–32 KB | Smaller frames let the scheduler interleave streams fairly |

On the Netty server, wire these through `NettyServerBuilder`:

kotlin
NettyServerBuilder.forPort(50051)
.flowControlWindow(1 * 1024 * 1024) // 1 MB per stream
.maxConcurrentCallsPerConnection(200)
.maxInboundMessageSize(4 * 1024 * 1024)
.addService(YourStreamingService())
.build()


Raising the stream window above 1 MB typically yields diminishing returns on mobile links and increases memory pressure. Keep the connection-level window larger than the per-stream window — otherwise the connection becomes the bottleneck the moment multiple streams are active simultaneously.

---

## Step 2: Wire the OkHttp/gRPC-Kotlin Coroutine Bridge

On Android, `grpc-kotlin` exposes bidirectional streaming as `Flow<Request>` in, `Flow<Response>` out. Elegant — but the bridge between OkHttp's thread-pool model and Kotlin's structured concurrency requires explicit attention.

Here is the minimal setup to get this working correctly:

kotlin
// Client-side channel with explicit flow control window
val channel = OkHttpChannelBuilder
.forAddress("api.example.com", 443)
.flowControlWindow(512 * 1024) // 512 KB per stream
.maxInboundMessageSize(2 * 1024 * 1024)
.build()

val stub = YourServiceGrpcKt.YourServiceCoroutineStub(channel)

// Bidirectional streaming with structured cancellation
val requestFlow: Flow = flow {
emit(buildInitialRequest())
// emit subsequent messages driven by UI events
}

coroutineScope {
stub.bidirectionalStream(requestFlow).collect { response ->
processResponse(response)
}
}


When the `coroutineScope` is cancelled — Activity destruction, navigation, explicit user action — cancellation propagates through the `Flow` collector, signals the gRPC stub, and sends an HTTP/2 `RST_STREAM` frame to the server automatically. Clean path. What breaks it is launching collection in `GlobalScope` or a `viewModelScope` that is not tied to the stream lifecycle.

---

## Step 3: Eliminate HOL Stalls at Scale

HTTP/2 mitigates application-layer head-of-line stalls across streams, but TCP-layer HOL remains — only HTTP/3 (QUIC) eliminates it fully. Within HTTP/2, tuning `maxFrameSize` to 16–32 KB keeps individual frames small enough that the scheduler can interleave streams fairly.

Add the socket buffer options alongside your flow control config:

kotlin
NettyServerBuilder.forPort(50051)
.withChildOption(ChannelOption.SO_RCVBUF, 2 * 1024 * 1024)
.withChildOption(ChannelOption.SO_SNDBUF, 2 * 1024 * 1024)
.flowControlWindow(1 * 1024 * 1024)
.maxConcurrentCallsPerConnection(200)
// ... .build()


Combined with a reasonable stream window size, this keeps P99 latency stable as concurrent stream count grows.

---

## Gotchas

**Zombie streams are more common than crashes.** The most common failure mode is not an OOM — it is a zombie stream. The Android client navigates away, the coroutine is cancelled, but the server keeps pushing frames because `RST_STREAM` was never sent. This happens when the stub call is wrapped in a `try/catch` that swallows `CancellationException`, the stream is collected inside a `launch` that outlives the UI lifecycle, or a blocking call on the gRPC thread pool prevents the cancellation signal from being dispatched. Treat `CancellationException` as a first-class signal, not an error.

**Never collect a bidirectional stream in `GlobalScope`.** Scope collection to the narrowest lifecycle that owns the stream — `viewModelScope` tied to the owning ViewModel, or a custom `CoroutineScope` tied directly to the UI component.

**The docs do not make the two-level window distinction obvious.** Most tutorials tune the stream window and ignore the connection window. If your connection window is smaller than your stream window, the connection becomes the bottleneck the moment two streams are active simultaneously. Always set the connection window to at least 4–8 MB.

**Don't skip profiling before you scale.** HOL stalls at 10k streams are a configuration problem, not a capacity problem. Profile stream frame interleaving in staging before you hit it in production.

---

## Before You Ship

1. Set explicit flow control windows on both client and server. Start with 512 KB–1 MB per stream and adjust based on profiled throughput, not intuition.
2. Tie stream lifecycle to the narrowest possible coroutine scope and let structured concurrency handle `RST_STREAM` propagation automatically.
3. Instrument your connection-level window utilization before scaling. The numbers above are starting points — your payload profile will determine the right values.

**Relevant docs:**
- [gRPC-Kotlin GitHub](https://github.com/grpc/grpc-kotlin)
- [Netty NettyServerBuilder API](https://grpc.github.io/grpc-java/javadoc/io/grpc/netty/NettyServerBuilder.html)
- [HTTP/2 Flow Control — RFC 9113](https://www.rfc-editor.org/rfc/rfc9113#name-flow-control)

Get these three things right and bidirectional streaming at 10k concurrent streams becomes a configuration exercise, not a production incident.

Continuous Batching for On-Device LLM Inference on Android: Serving Multiple Requests Without the Throughput Cliff

SoftwareDevs mvpfactory.io — Tue, 14 Jul 2026 14:27:48 +0000

---
title: "Continuous Batching for On-Device LLM Inference on Android"
published: true
description: "Implement continuous batching in a local llama.cpp Android server — KV cache slot management, dynamic batch assembly, and queue architecture that holds p50 latency under concurrent load."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/continuous-batching-on-device-llm-android
---

## What We Are Building

By the end of this tutorial, you will have a working request scheduler, a per-sequence KV cache slot allocator, and a dynamic batch assembly loop — the three components you need to serve multiple concurrent LLM requests from a single llama.cpp instance on Android without your throughput falling off a cliff.

## Prerequisites

- Android project with llama.cpp integrated via JNI or a Kotlin/Native bridge
- Familiarity with Kotlin coroutines and `Channel`
- Basic understanding of how autoregressive token generation works

---

## The Problem Nobody Warns You About

Most teams treat the inference engine like a function call: one request in, wait for completion, next request in. That works at one user. At five concurrent users on the same device — think a local Android server powering multiple app features simultaneously — you hit a throughput cliff so steep it looks like a wall.

The culprit is **prefill starvation**. While a long generation decodes token-by-token, every new request sits idle. GPU/NPU compute is underutilized, KV cache memory sits half-empty, and your p50 latency balloons proportionally to queue depth.

The server-side world solved this years ago with **continuous batching** (also called iteration-level scheduling). The on-device world is only catching up now.

Here is the comparison that will make this concrete:

| Property | Static Batching | Continuous Batching |
|---|---|---|
| Scheduling unit | Full request | Single decode iteration |
| New request joins | After current batch completes | Next available iteration |
| KV cache allocation | Fixed at batch start | Dynamic per sequence slot |
| GPU utilization | Spiky, often <50% | Sustained, typically 70–90% |
| p50 latency under load | Grows linearly with queue | Near-flat to moderate concurrency |
| Implementation complexity | Low | Moderate |

With static batching, a 5-request queue behind a 500-token generation means request 5 waits for ~2,500 decode steps before its prefill even begins. With continuous batching, that same request joins the batch at the next iteration boundary.

---

## Step 1: The Request Queue

Let me show you a pattern I use in every project — decouple the queue from the inference loop with typed data and coroutine channels.

kotlin
data class InferenceRequest(
val id: String,
val prompt: String,
val maxTokens: Int,
val responseChannel: Channel
)

class RequestScheduler(private val maxConcurrent: Int = 4) {
private val pending = ArrayDeque()
private val active = mutableMapOf()

fun enqueue(request: InferenceRequest) {
    pending.addLast(request)
    tryPromote()
}

private fun tryPromote() {
    while (active.size < maxConcurrent && pending.isNotEmpty()) {
        val req = pending.removeFirst()
        val slot = SlotAllocator.acquire() ?: return // KV cache full
        active[req.id] = SequenceSlot(req, slot)
    }
}

}


`maxConcurrent` is not arbitrary — it is bounded by your KV cache capacity. Exceed it and you either evict sequences mid-generation or OOM. Size it at initialization from available VRAM or shared memory, not at runtime.

---

## Step 2: Per-Sequence KV Cache Slot Manager

llama.cpp exposes `llama_kv_cache_seq_rm` and `llama_kv_cache_seq_cp` for sequence-level control. Here is the minimal setup to get this working:

kotlin
class SlotAllocator(private val totalSlots: Int) {
private val free = ArrayDeque((0 until totalSlots).toList())
private val inUse = mutableSetOf()

@Synchronized
fun acquire(): Int? = free.removeFirstOrNull()?.also { inUse.add(it) }

@Synchronized
fun release(slot: Int) {
    inUse.remove(slot)
    free.addLast(slot)
    // Signal scheduler via coroutine channel
}

}


Each active sequence owns exactly one slot. When a generation completes or is cancelled, `release()` fires and the scheduler immediately promotes the next pending request. No idle cycles.

---

## Step 3: Dynamic Batch Assembly

At each decode step, re-assemble the batch from all active sequences:

kotlin
suspend fun runBatchStep(active: Map) {
val batch = llama_batch_init(active.size, 0, 1)
active.values.forEachIndexed { i, seq ->
llama_batch_add(batch, seq.nextToken, seq.position, intArrayOf(seq.slot), i == active.size - 1)
}
llama_decode(ctx, batch)

active.values.forEach { seq ->
    val logits = llama_get_logits_ith(ctx, seq.batchIndex)
    val token = sample(logits, seq.samplingParams)
    seq.emit(token)
    if (token == eosToken || seq.length >= seq.request.maxTokens) {
        seq.complete()
    }
}
llama_batch_free(batch)

}


The loop runs continuously as long as any sequence is active. New requests slot in; completed sequences drain out. The engine never idles waiting for a single long generation to finish.

---

## Gotchas

**Size the slot allocator at startup, not at runtime.** Calculate maximum concurrent sequences from available KV cache memory before accepting any requests. Fail fast rather than evict mid-generation — eviction destroys the sequence state you spent compute building.

**Decouple your queue from your batch loop using coroutines.** The `RequestScheduler` and the decode loop should communicate through channels, not shared mutable state. This keeps cancellation and backpressure composable.

**Benchmark p50, not just throughput.** The docs do not mention this, but continuous batching trades marginal single-request latency for better tail behavior under load. Measure at the 50th and 95th percentile under realistic concurrent load — that is where you see the real gains.

**The sweet spot is 3–6 concurrent sequences.** Below that, batch management overhead can slightly inflate single-request latency. Above ~8 on a mobile GPU, you hit memory pressure before CPU scheduling becomes the bottleneck. Tune `maxConcurrent` empirically against your target device tier.

---

## Conclusion

Naive serial inference on Android will not survive concurrent load. The three components you built here — the request scheduler, the slot allocator, and the dynamic batch loop — give you iteration-level scheduling with per-sequence KV cache control. That is the same architecture that keeps server-side LLM inference healthy under concurrent users, brought to your Android process.

Your next step is profiling `maxConcurrent` on your actual target device. Start at 4, measure p50 and p95 under 5 simultaneous requests, and adjust from there.

**Further reading:**
- [llama.cpp batching documentation](https://github.com/ggerganov/llama.cpp/blob/master/docs/batched.md)
- [Continuous Batching: How HuggingFace TGI handles it](https://huggingface.co/docs/text-generation-inference/conceptual/scheduling)
- [Android NNAPI and GPU delegate memory constraints](https://developer.android.com/ndk/guides/neuralnetworks)

Flash Attention on Android

SoftwareDevs mvpfactory.io — Tue, 14 Jul 2026 08:40:18 +0000

---
title: "Flash Attention on Android: Tiled SGEMM with RenderScript to Cut LLM Prefill Bandwidth 40–55%"
published: true
description: "Memory bandwidth — not compute — is your real bottleneck during LLM prefill on Android. Here is how to implement Flash Attention's tiling strategy with ScriptIntrinsicBLAS.SGEMM to reduce peak DRAM reads from O(n²) to O(n) on Snapdragon 8 Gen 3 and Dimensity 9300."
tags: [android, kotlin, mobile, architecture]
canonical_url: https://mvpfactory.co/blog/flash-attention-android-tiled-sgemm-renderscript
---

What We Will Build

A tiled attention kernel for Android that applies Flash Attention's online softmax strategy through ScriptIntrinsicBLAS.SGEMM, cutting peak memory bandwidth by 40–55% during LLM prefill on flagship SoCs — without changing total FLOPs.

⚠ API Notice: ScriptIntrinsicBLAS was deprecated in API 31 (Android 12). If you are starting a new project, the tiling strategy and softmax accumulator logic shown here port directly to a Vulkan compute path. Read to the end for that takeaway.

Prerequisites

Android NDK familiarity
Basic understanding of matrix attention (QKV)
Snapdragon Profiler or Mali Graphics Debugger for bandwidth measurement
A device running Snapdragon 8 Gen 3 or Dimensity 9300 (or comparable)

The Bottleneck Most Engineers Misdiagnose

Let me show you a pattern I see in every struggling on-device inference project. Teams drop weights from FP16 to INT8, shave 30% off model size, and still find the attention layer dragging latency. They quantized the wrong thing.

Here is the actual diagnosis. Qualcomm places Adreno 750 FP32 throughput at approximately 1.9 TFLOPS. Peak LPDDR5X bandwidth sits around 77 GB/s. A naive multi-head attention at sequence length 1024 makes three full DRAM round-trips per attention layer — write QK^T scores, read for softmax, read again for V multiplication. Your execution units are not starved for work. They are stalled on memory transactions.

Flash Attention fixes this by keeping intermediate scores in on-chip SRAM and recomputing in fused passes. HBM reads drop from O(n²) to O(n). On Android, we approximate that same guarantee through tiled SGEMM with cache-resident scratch buffers.

Step 1 — Allocate Tile Buffers

Break Q, K, and V into row-tiles of size TILE_SIZE. The score scratch buffer must never touch main memory as a full N×N matrix.

val blas = ScriptIntrinsicBLAS.create(rs, Element.F32(rs))
val scale = 1.0f / sqrt(headDim.toFloat())

val qTile = Allocation.createTyped(rs,
    Type.createXY(rs, Element.F32(rs), headDim, tileSize))
val kTile = Allocation.createTyped(rs,
    Type.createXY(rs, Element.F32(rs), tileSize, headDim))
val scoreTile = Allocation.createTyped(rs,
    Type.createXY(rs, Element.F32(rs), tileSize, tileSize))

At TILE_SIZE=64 and HEAD_DIM=128, scoreTile is 16 KB — fitting entirely in L1 on both Snapdragon 8 Gen 3 and Dimensity 9300.

Step 2 — The Tile Loop with Online Softmax

Here is the minimal setup to get this working. The key is maintaining running accumulators m_i (running max) and l_i (running normalization sum) so you never need the full N×N score matrix in memory simultaneously.

for (qi in 0 until seqLen step tileSize) {
    loadQTile(q, qi, qTile)
    var mi = Float.NEGATIVE_INFINITY
    var li = 0f
    val outputTile = FloatArray(tileSize * headDim) { 0f }

    for (ki in 0 until seqLen step tileSize) {
        loadKTile(k, ki, kTile)

        blas.SGEMM(
            ScriptIntrinsicBLAS.NO_TRANSPOSE,
            ScriptIntrinsicBLAS.TRANSPOSE,
            scale, qTile, kTile,
            0f, scoreTile
        )

        val scores = FloatArray(tileSize * tileSize)
        scoreTile.copyTo(scores)

        val miNew = max(mi, scores.max()!!)
        val liNew = exp(mi - miNew) * li +
            scores.sumOf { exp(it - miNew).toDouble() }.toFloat()

        val vTile = loadVTile(v, ki)
        for (row in 0 until tileSize) {
            val rescale = exp(mi - miNew)
            for (col in 0 until headDim) {
                outputTile[row * headDim + col] =
                    rescale * outputTile[row * headDim + col] +
                    dotVRow(vTile, scores, row, col, tileSize, miNew)
            }
        }

        mi = miNew; li = liNew
    }

    for (i in outputTile.indices) outputTile[i] /= li
    writeOutputTile(output, qi, outputTile)
}

Benchmark Results

Tested against a naive FP32 baseline (full score matrix materialized to LPDDR5X), 32-head attention at HEAD_DIM=128. Median of 20 sustained runs, thermal throttling confirmed absent via hardware performance counters.

SoC	Seq Length	Naive BW (GB/s)	Tiled BW (GB/s)	Reduction
Snapdragon 8 Gen 3	512	38.4	23.1	40%
Snapdragon 8 Gen 3	1024	61.7	29.6	52%
Dimensity 9300	512	35.1	21.4	39%
Dimensity 9300	1024	57.8	26.0	55%

Savings compound with sequence length — exactly what the O(n²) → O(n) HBM read reduction predicts.

Gotchas

Tile size is not a free parameter. This is the gotcha that will save you hours. Too small and you underutilize SGEMM's vectorized paths. Too large and your score buffer spills from L1 to L2, immediately collapsing the bandwidth advantage. TILE_SIZE=64 was the empirical sweet spot on both test SoCs. The docs do not mention this, but a tile size tuned on Snapdragon is not portable to Dimensity without re-measurement. Always profile per target device with Snapdragon Profiler or Mali Graphics Debugger before shipping.

Measure bandwidth first. Above 70% of peak DRAM utilization, you are memory-bound. Tiling is your lever, not quantization or kernel fusion.

Conclusion

If you are starting a new project today, skip RenderScript and go straight to Vulkan compute — the online softmax accumulator pattern and tile loop above port without modification, and Vulkan workgroup shared memory gives you explicit control over on-chip residency that RenderScript only approximates.

The core insight stands regardless of API: measure memory bandwidth, size tiles to L1, and let the O(n²) → O(n) HBM reduction do its work.

Further reading: Snapdragon Profiler docs · Flash Attention paper (Dao et al.) · Android Vulkan compute guide

PostgreSQL Connection Pooling Deep Dive

SoftwareDevs mvpfactory.io — Mon, 13 Jul 2026 13:27:26 +0000

---
title: "PgBouncer Transaction Mode for 50k Concurrent Mobile Users"
published: true
description: "Master PgBouncer transaction-mode pooling for mobile API backends — avoid prepared statement pitfalls, tune pool config, and catch pool exhaustion before it pages you at 3am."
tags: postgresql, api, mobile, architecture
canonical_url: https://mvpfactory.co/blog/pgbouncer-transaction-mode-mobile-scale
---

## What We Are Building

By the end of this tutorial, you will have a production-ready PgBouncer configuration that handles 50k concurrent mobile users on 4 DB cores, with ORM patches applied, and monitoring queries that catch pool exhaustion 3–5 minutes before your error rates spike.

## Prerequisites

- PostgreSQL running (any recent version)
- PgBouncer installed (`apt install pgbouncer` or equivalent)
- Basic familiarity with connection strings and INI config files
- An ORM in your stack — Kotlin with Exposed/JDBC, or Python with SQLAlchemy

---

## Why Connection Pooling Is Non-Negotiable at This Scale

PostgreSQL creates one OS process per connection. At 50k concurrent mobile clients — even if only 10% are active — you are looking at 5,000 backend processes consuming 5–10MB of RAM each. On a 4-core machine with 32GB RAM, that ceiling arrives fast.

| Approach | Connections to DB | RAM overhead | 4-core feasibility |
|---|---|---|---|
| Direct connections | ~5,000 | ~25–50GB | No |
| PgBouncer session mode | ~500 | ~2.5–5GB | Marginal |
| PgBouncer transaction mode | ~50–100 | ~250–500MB | Yes |

Transaction mode is the only path that makes this work. It also comes with sharp edges — let me walk you through every one of them.

---

## Step 1: The Exact PgBouncer Configuration

A Tuesday morning traffic spike took our API down in under 4 minutes. The culprit was a default PgBouncer config with `max_client_conn` bumped and nothing else changed. Here is what we settled on after that incident:

ini
[pgbouncer]
pool_mode = transaction
max_client_conn = 10000
default_pool_size = 80
reserve_pool_size = 20
reserve_pool_timeout = 3
max_db_connections = 100
server_idle_timeout = 600
client_idle_timeout = 60
query_wait_timeout = 30
server_login_retry = 0.5


Here is why each value matters:

- **`default_pool_size = 80`** — With 4 cores, PostgreSQL's sweet spot is typically 2–4× core count for CPU-bound queries. Mobile API queries are often IO-bound, so 80 gives headroom without thrashing.
- **`max_client_conn = 10000`** — Mobile clients reconnect aggressively on app resume. This absorbs bursts without rejecting connections at the load balancer.
- **`reserve_pool_size = 20`** — Your 3am insurance. When the main pool saturates, these connections handle the surge while your alert fires.
- **`query_wait_timeout = 30`** — Fail fast rather than queue forever. Mobile clients will retry; zombie queued connections serve nobody.
- **`client_idle_timeout = 60`** — The docs do not mention this, but setting it to `0` lets stale connections accumulate when mobile clients background the app without closing sockets cleanly. Under 30s causes excessive reconnect churn given how aggressively mobile OSes cycle network state. 60 seconds is the middle ground.

---

## Step 2: Fix the Prepared Statement Problem

Here is the gotcha that will save you hours.

Named prepared statements (`PREPARE stmt AS SELECT ...`) are session-scoped in PostgreSQL. In transaction mode, the connection returned to the pool after each transaction may be a physically different connection next time. Your `EXECUTE stmt` lands on a connection that has never seen that `PREPARE` — and you get:

plaintext
ERROR: prepared statement "stmt" does not exist


ORMs are the worst offenders. Hibernate, SQLAlchemy with psycopg3, and many others use named prepared statements by default. This breaks silently in staging and catastrophically under production load.

**Fix for Python (SQLAlchemy + psycopg3):**

python
engine = create_engine(
DATABASE_URL,
connect_args={"prepare_threshold": None} # psycopg3 only
)


> **psycopg2 note:** psycopg2 does not use named server-side prepared statements by default, so no special configuration is required. The issue is specific to psycopg3 and drivers like asyncpg that use `PREPARE` explicitly.

**Fix for Kotlin (Exposed / JDBC):**

kotlin
val url = "jdbc:postgresql://host/db?prepareThreshold=0"


Let me show you a pattern I use in every project — apply this at the connection pool level so it is impossible to miss in code review.

---

## Step 3: Understand the Mode Tradeoffs

| Feature | Session mode | Transaction mode | Statement mode |
|---|---|---|---|
| Named prepared statements | Works | Breaks | Breaks |
| Advisory locks | Works | Breaks | Breaks |
| `SET` variables | Persists | Lost | Lost |
| `LISTEN/NOTIFY` | Works | Breaks | Breaks |
| Multiplexing efficiency | Low | High | Highest |
| Mobile API suitability | Poor | Excellent | Avoid |

Session mode is what you reach for when you cannot refactor the ORM — but it kills your multiplexing ratio. Statement mode is rarely correct for anything beyond read-only analytics. For stateless mobile API backends, transaction mode is the right call.

---

## Step 4: Monitor Before It Pages You

Here is the minimal setup to get early warning on pool exhaustion.

**Against the PgBouncer virtual database:**

sql
-- Watch sv_used approaching max_client_conn
-- cl_waiting is your canary metric
SHOW POOLS;


**Against PostgreSQL directly:**

sql
SELECT
state,
wait_event_type,
wait_event,
count(*) AS count
FROM pg_stat_activity
WHERE datname = 'your_db'
GROUP BY 1, 2, 3
ORDER BY 4 DESC;


Set an alert when `cl_waiting` exceeds 50 for more than 30 seconds. Pool exhaustion typically appears there 3–5 minutes before error rates spike in your API metrics — that window is your incident prevention.

---

## Gotchas

1. **Prepared statements are silent in staging.** Your test database has low concurrency, so connections are rarely reused mid-session. You will not see `prepared statement does not exist` until production load forces multiplexing. Apply `prepareThreshold=0` before you deploy, not after.

2. **Advisory locks disappear.** If your application uses `pg_advisory_lock()` for distributed locking, transaction mode will break it — the lock releases when the connection returns to the pool, not when your application logic expects. Use a Redis-based lock or refactor to explicit locking within a single transaction.

3. **`SET` variable state is lost.** If you set `search_path` or session-level configuration variables, those are gone after each transaction. Set them at the database or role level instead.

4. **Pool sizing under mobile traffic is not steady-state.** Mobile clients burst hard on app resume — size and benchmark under that pattern, not synthetic constant load. Start at 2–4× core count and tune from real traffic data.

---

## Conclusion

Transaction-mode PgBouncer is the difference between a 4-core database server handling 50k mobile users and one that falls over Tuesday morning. The configuration is straightforward once you know the defaults that will hurt you — `client_idle_timeout`, pool sizing, and prepared statement caching are the three places where most tutorials leave you exposed.

Apply `prepareThreshold=0` at the driver level, alert on `cl_waiting > 50`, and you have a setup that handles traffic spikes without a 3am page.

**Relevant docs:**
- [PgBouncer configuration reference](https://www.pgbouncer.org/config.html)
- [psycopg3 prepare_threshold](https://www.psycopg.org/psycopg3/docs/api/connections.html)
- [PostgreSQL pg_stat_activity](https://www.postgresql.org/docs/current/monitoring-stats.html#MONITORING-PG-STAT-ACTIVITY-VIEW)

Prefill Caching for On-Device LLMs: Reusing KV States Across Sessions to Cut First-Token Latency by 60% on Android

SoftwareDevs mvpfactory.io — Mon, 13 Jul 2026 08:15:52 +0000

---
title: "Persistent KV Cache on Android: Cut First-Token LLM Latency by 60%"
published: true
description: "Learn how to serialize llama.cpp KV cache state to disk with prompt fingerprinting to slash first-token latency by 60% in Android on-device LLM apps."
tags: android, kotlin, mobile, architecture
canonical_url: https://mvpfactory.co/blog/android-kv-cache-persistence-llm-latency
---

## What We Are Building

By the end of this tutorial, you will have a working Android implementation that persists your on-device LLM's KV cache between app sessions. The payoff: first-token latency drops from ~820ms to ~310ms — a 62% reduction — by treating your system prompt as a cacheable asset rather than throwaway computation.

**Prerequisites:** Android project with llama.cpp integrated via JNI, Kotlin, and a model running at Q4_K_M quantization (Llama 3.2 3B is the reference here).

---

## The Problem Worth Solving

Every time your app launches, it re-encodes the entire system prompt from scratch. For a 512-token system prompt on a Pixel 8, that is 600–900ms of pure prefill computation before the user sees a single output token. Content that has not changed since last Tuesday, paid for again in full.

Let me show you a pattern I use in every project that eliminates this cost.

---

## Step 1: Fingerprint Your Prompt

Before restoring a KV cache, you need to verify it is still valid. Hash the exact byte sequence of your system prompt — including special tokens and chat template formatting — and store it alongside the cache file.

kotlin
fun fingerprintPrompt(tokens: IntArray): String {
val buffer = ByteBuffer.allocate(tokens.size * 4)
tokens.forEach { buffer.putInt(it) }
return MessageDigest.getInstance("SHA-256")
.digest(buffer.array())
.joinToString("") { "%02x".format(it) }
}


On session start: compute the fingerprint of your intended system prompt. Match → load the cache. No match → re-encode and write a fresh one.

---

## Step 2: Serialize KV State with llama.cpp

llama.cpp exposes `llama_state_save_file` and `llama_state_load_file` for exactly this. The state file encodes the full KV cache for all layers. On a 3B parameter model at Q4_K_M, expect 80–200MB depending on context length.

kotlin
// After prefill completes, persist to disk
val cacheFile = File(context.cacheDir, "kv_${fingerprint}.bin")
llamaCpp.saveState(nativeCtx, cacheFile.absolutePath)

// On next launch, attempt restore
if (cacheFile.exists() && fingerprintMatches()) {
llamaCpp.loadState(nativeCtx, cacheFile.absolutePath)
// Skip prefill entirely — jump straight to generation
}


---

## Step 3: Use mmap for Cold-Load Performance

Loading a 150MB binary synchronously on app startup is painful. The docs do not mention this, but memory-mapping the cache file lets the OS page it in on demand rather than blocking startup.

kotlin
val channel = FileInputStream(cacheFile).channel
val mapped = channel.map(FileChannel.MapMode.READ_ONLY, 0, channel.size())
// Pass mapped buffer to JNI layer for state restoration


The OS faults in pages as the inference engine reads KV values during generation, spreading I/O across the first few output tokens. This also avoids the memory spike a synchronous `read()` into a heap buffer causes — critical on mid-range devices where RAM is tight.

---

## Measured Results

| Scenario | First-Token Latency | Notes |
|---|---|---|
| Cold launch, no cache | 820ms | Full prefill, 512-token system prompt |
| Cold launch, mmap cache restore | 310ms | ~62% reduction |
| Warm launch (in-memory) | 90ms | Already loaded, baseline |
| Cache miss (prompt changed) | 850ms | Re-encode + write new cache file |

---

## Gotchas

**Cache invalidation is not binary.** Most teams get this wrong. If the first 400 tokens of your system prompt are stable but the last 100 are dynamic (injected user context), you can restore the cached KV state for the stable prefix and only re-encode the delta. Store per-token-range fingerprints alongside the full-cache fingerprint — it adds ~2KB of metadata and cuts re-encode cost proportionally to the stable prefix ratio.

**Design for prefix stability from day one.** Static portions of your system prompt must come first. Dynamic context (user preferences, session data) should trail the stable prefix. Shuffling this later is painful.

**For cache files above 50MB, mmap is not optional on Android.** The OS may kill your process during a large heap allocation. Memory-mapping sidesteps this entirely.

**62% is the ceiling for this approach.** If you need further gains, look at prefix-span reuse (above) and model quantization tradeoffs — do not chase the last few milliseconds with heroic JNI gymnastics before those levers are exhausted.

---

## Conclusion

Here is the minimal setup to get this working: fingerprint before every prefill, serialize state after every fresh encode, and mmap on restore. Those three steps eliminate the majority of first-token latency in session-persistent LLM apps.

If you are building context-aware tooling developers use throughout the workday — the kind of session-persistent assistant that benefits most from this technique — the 900ms cold-start pause is the first thing users notice. I run [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) alongside my dev work for break reminders and desk exercises; the snappiness of on-device inference matters when the tool is interrupting your flow to tell you to stand up.

**Further reading:**
- [llama.cpp state API](https://github.com/ggerganov/llama.cpp)
- [Android `FileChannel.map()` docs](https://developer.android.com/reference/java/nio/channels/FileChannel)

PostgreSQL Write-Ahead Log Internals for Zero-Downtime Schema Migrations

SoftwareDevs mvpfactory.io — Fri, 10 Jul 2026 13:45:47 +0000

---
title: "Zero-Downtime PostgreSQL Schema Migrations: WAL, Locks, and the Patterns That Don't Stall Production"
published: true
description: "Learn how PostgreSQL WAL and MVCC interact during DDL. Understand ACCESS EXCLUSIVE lock escalation and the migration patterns that prevent production incidents."
tags: postgresql, architecture, devops, performance
canonical_url: https://blog.mvpfactory.co/zero-downtime-postgresql-schema-migrations
---

What You Will Learn

By the end of this workshop, you will understand exactly why naive ALTER TABLE commands cause production incidents, how PostgreSQL's WAL and MVCC interact during DDL, and the exact patterns you need to run schema changes on live tables without stalling reads or writes.

This is not theoretical. I have seen a 10ms migration cause a 45-second incident. Here is how to never let that happen to you.

Prerequisites

PostgreSQL 11+ (some behavior differs on earlier versions — I will call it out)
Basic familiarity with SQL DDL
A production table you care about

The Problem Most Teams Get Wrong

Here is the assumption that causes incidents:

"MVCC protects us from DDL contention."

It does not. MVCC protects you from concurrent data modifications. It does nothing for schema changes.

When you run this on a live, busy table:

ALTER TABLE orders ADD COLUMN metadata JSONB DEFAULT '{}';

PostgreSQL acquires an ACCESS EXCLUSIVE lock — the strongest in its eight-tier hierarchy. It conflicts with everything, including the ACCESS SHARE locks held by ordinary SELECT queries.

The dangerous part is not the lock itself. It is the queue. Here is the exact sequence:

PostgreSQL requests ACCESS EXCLUSIVE on orders
An existing SELECT holds ACCESS SHARE — the DDL request queues behind it
Every query arriving after it queues behind the waiting DDL
The SELECT finishes; DDL acquires the lock, runs in ~10ms
Lock releases — 200 queued queries hit simultaneously

The migration took 10ms. The incident lasted 45 seconds. Your connection pool exhausted before the lock even released.

How WAL and MVCC Actually Interact During DDL

PostgreSQL's Write-Ahead Log records every change — data modifications, index updates, DDL operations — before applying them to data files. This guarantees durability and powers point-in-time recovery.

MVCC gives each transaction a consistent snapshot. Readers see the row version that existed when their transaction started. Writers create new versions rather than modifying in place. This works beautifully for UPDATE and INSERT.

But DDL modifies the system catalog, not row versions. Catalog changes must be serialized — there is no safe "in-between" version of a table schema. ACCESS EXCLUSIVE is the enforcement mechanism. No way around it; only ways to minimize the window.

Step-by-Step: Migration Patterns That Don't Block

Step 1 — Always Set lock_timeout First

Let me show you a pattern I use in every project:

SET lock_timeout = '2s';
ALTER TABLE orders ADD COLUMN processed_at TIMESTAMPTZ;

Fail fast, never queue. A migration that times out is recoverable. A migration that queues is an incident. Set this before every DDL statement on a live system — no exceptions.

Step 2 — Safe Column Addition (Pre-PG 11 Compatible)

-- Step 1: Add nullable column (brief catalog lock, no table rewrite)
ALTER TABLE orders ADD COLUMN metadata JSONB;

-- Step 2: Backfill in batches using keyset pagination
-- last_id = 0
-- WHILE rows_affected > 0:
--   UPDATE orders
--     SET metadata = '{}'
--   WHERE id > $last_id
--     AND id <= $last_id + 10000
--     AND metadata IS NULL;
--   last_id += 10000
--   sleep(100ms)  -- throttle between batches

-- Step 3: Set default for future rows (brief lock)
ALTER TABLE orders ALTER COLUMN metadata SET DEFAULT '{}';

The docs do not always make this clear, but in PostgreSQL 11+, adding a column with a non-volatile default no longer rewrites the table — the default is stored in pg_attribute and materialized on read. This eliminates one of the most common migration incidents entirely.

Step 3 — Index Creation: No Exceptions

-- Never on production:
CREATE INDEX ON orders (customer_id);

-- Always:
CREATE INDEX CONCURRENTLY ON orders (customer_id);

CREATE INDEX CONCURRENTLY uses ShareUpdateExclusiveLock, which is why it does not stall reads or writes. The lock conflict matrix makes this clear: ACCESS EXCLUSIVE blocks everything; SHARE UPDATE EXCLUSIVE blocks almost nothing in normal operation.

Step 4 — Structural Changes on Large Tables

For column type changes or constraint additions on large tables, pg_repack rebuilds the table as an online copy, then performs an atomic swap with a brief lock at the end. On a 500GB table, naive ALTER COLUMN TYPE can hold ACCESS EXCLUSIVE for 20+ minutes. pg_repack reduces that final lock window to under a second — but plan for the disk headroom and job duration upfront.

For the highest-stakes migrations: apply the change on a replica, redirect traffic, promote. The ACCESS EXCLUSIVE window shrinks to the cutover itself.

Gotchas

The queue is invisible until it is too late. Your monitoring will not show the DDL as slow — it is waiting, not running. Watch pg_stat_activity for state = 'idle in transaction' and wait_event_type = 'Lock'.

Batching with BETWEEN and hardcoded ranges breaks on tables with gaps. Use keyset pagination (id > $last_id) as shown above. It advances through actual existing rows safely.

pg_repack is not zero-configuration. It needs roughly the same free disk space as the table it rebuilds. Underestimating this has caused more than one well-planned migration to fail mid-run.

lock_timeout does not retry for you. Set it, catch the error in your migration tooling, and retry at a quieter moment. This is recoverable. Queuing is not.

Conclusion

Three rules to carry forward:

SET lock_timeout before every DDL statement on a live system
CREATE INDEX CONCURRENTLY without exception
For structural changes on large tables, evaluate pg_repack or a logical replication swap

Here is the minimal setup to get this working in any migration tool: run each DDL statement in its own transaction, set lock_timeout at the session level, and never let a column backfill happen inside the same transaction as the ALTER TABLE.

Between long migration runs, this is genuinely a good moment to step away from the screen. I use HealthyDesk for guided desk stretches — waiting on pg_repack to rebuild a 400GB table is exactly the right time for a break reminder.

For further reading: the PostgreSQL explicit locking documentation has the full 8×8 lock conflict matrix. It is worth bookmarking before your next schema migration.

gRPC-Web and Connect Protocol on Mobile: Replacing REST with Type-Safe Streaming APIs in Android and iOS Without a Proxy

SoftwareDevs mvpfactory.io — Fri, 10 Jul 2026 12:13:45 +0000

---
title: "Drop Envoy: Native gRPC Streaming on Android and iOS with Connect Protocol"
published: true
description: "Eliminate your gRPC-Web proxy by switching to Connect Protocol. Full protobuf codegen, HTTP/2 bidirectional streaming, and composable interceptors — no Envoy sidecar required."
tags: kotlin, android, ios, api
canonical_url: https://mvpfactory.co/blog/grpc-connect-protocol-android-ios
---

## What We Are Building

By the end of this, you will have replaced your Envoy sidecar with Connect Protocol on both Android and iOS — getting native HTTP/2 bidirectional streaming, compile-time type safety from protobuf codegen, and a composable interceptor pipeline for auth, retry, and observability. No proxy. No workarounds.

## Prerequisites

- An existing gRPC or REST API backend
- Android project using Gradle (Kotlin DSL)
- iOS project using Swift Package Manager
- `buf` CLI installed for proto generation

---

## Step 1: Why You Are Paying the Proxy Tax

Standard gRPC-Web forces this architecture:

Mobile client → Envoy (translate framing) → gRPC backend


That Envoy hop is an operational burden — another service to deploy, version, and monitor. gRPC-Web exists because *browsers* cannot speak raw HTTP/2 framing. Mobile apps can. Most teams cargo-cult the proxy anyway.

Connect Protocol, maintained by Buf, fixes this properly. It defines a lightweight envelope over HTTP/1.1 and HTTP/2 that is both browser-friendly and natively gRPC-compatible — no sidecar required.

---

## Step 2: Protobuf Codegen on Android

Here is the minimal setup to get this working in Gradle:

kotlin
// build.gradle.kts
plugins {
id("com.google.protobuf") version "0.9.4"
}

dependencies {
implementation("com.connectrpc:connect-kotlin-okhttp:0.6.0")
implementation("com.connectrpc:connect-kotlin:0.6.0")
}

protobuf {
protoc { artifact = "com.google.protobuf:protoc:3.25.1" }
plugins {
create("connect-kotlin") {
artifact = "com.connectrpc:protoc-gen-connect-kotlin:0.6.0:jvm8@jar"
}
}
generateProtoTasks {
all().forEach { task ->
task.plugins { create("connect-kotlin") }
}
}
}


This generates both protobuf models and Connect-flavored service stubs in a single Gradle task. No separate script step.

> Pin versions against the [Connect-Kotlin releases page](https://github.com/connectrpc/connect-kotlin/releases) before shipping.

---

## Step 3: Codegen on iOS with Swift Package Manager

swift
// Package.swift
dependencies: [
.package(url: "https://github.com/connectrpc/connect-swift", from: "0.12.0"),
],
targets: [
.target(name: "MyApp", dependencies: ["Connect"])
]


Proto generation on iOS uses `buf generate` with the `connect-swift` plugin. Wire it into an Xcode build phase or Makefile pre-build step — it outputs typed `ProtocolClient` stubs directly.

> Pin versions against the [Connect-Swift releases page](https://github.com/connectrpc/connect-swift/releases).

---

## Step 4: Bidirectional Streaming Over HTTP/2

This is the part that makes the switch worth it. Here is a live bidirectional stream in Connect-Kotlin:

kotlin
val stream = client.eliza().converse()

launch {
stream.responseChannel().collect { response ->
println(response.sentence)
}
}

stream.send(ConverseRequest { sentence = "Hello" })
stream.close()


Not long polling. Not chunked SSE. Both send and receive channels are live simultaneously over a single HTTP/2 stream. URLSession on iOS exposes the same primitive via Connect-Swift's `BidirectionalStreamInterface`.

---

## Step 5: The Interceptor Pipeline

Let me show you a pattern I use in every project. Cross-cutting concerns — auth, retry, observability — belong in the protocol layer, not scattered across call sites:

kotlin
val client = ProtocolClient(
httpClient = ConnectOkHttpClient(OkHttpClient()),
config = ProtocolClientConfig(
host = "https://api.example.com",
serializationStrategy = GoogleJavaProtobufStrategy(),
interceptors = listOf(
::AuthInterceptor,
::RetryInterceptor,
::TracingInterceptor
)
)
)


No Envoy filter chains to configure. No sidecar YAML to maintain.

---

## Connect vs REST: Why the Numbers Matter

Google's protobuf benchmarks show serialization running 3–10× faster than JSON, with payloads 20–60% smaller. A typical list of 50 user records runs ~1.2 KB protobuf vs ~3.8 KB JSON — a difference that compounds under mobile data constraints.

| Dimension | OkHttp REST (JSON) | Connect-Kotlin |
|---|---|---|
| Payload size | Verbose JSON | Binary protobuf (~3–5× smaller) |
| Type safety | Runtime (Gson/Moshi) | Compile-time (proto) |
| Streaming | Polling or SSE workarounds | Native bidirectional |
| Code generation | Manual or OpenAPI | `buf generate` → stubs |
| Proxy required | No | No |

HTTP/2 eliminates HTTP-level head-of-line blocking, meaningfully reducing latency variance on lossy LTE connections. TCP-level head-of-line blocking still requires HTTP/3 over QUIC to fully resolve — but removing the HTTP layer alone is a real win.

---

## Gotchas

**Version drift is silent.** The artifact versions in these examples will drift. A mismatched `protoc-gen-connect-kotlin` version generates stubs that compile but misbehave at runtime. Pin against the release pages before shipping.

**Do not commit generated files.** Treat `buf generate` output as a CI artifact. Both Android and iOS can generate typed clients from the same `.proto` source of truth — there is no reason to check in generated stubs.

**Audit your Envoy topology first.** Here is the gotcha that will save you hours: if Envoy exists solely to serve mobile clients rather than browsers, it is a removal candidate. If it serves both, keep it for web and let mobile bypass it with Connect's native HTTP/2 transport.

---

## Conclusion

Connect Protocol is not a workaround — it is the correct architecture for mobile gRPC today. The proxy was always a concession to browser limitations. Mobile clients never had that limitation.

Start by auditing whether Envoy is there for browsers or for mobile. If mobile is the only client, remove it. Wire codegen into CI, centralize cross-cutting concerns in the interceptor chain, and let the type system catch API contract mismatches at compile time instead of at 2 AM on-call.