DEV Community: SoftwareDevs mvpfactory.io

Speculative Decoding on Mobile GPUs

SoftwareDevs mvpfactory.io — Fri, 19 Jun 2026 14:20:55 +0000

---
title: "Speculative Decoding on Mobile GPUs: Draft-Verify LLM Pipelines with Vulkan Compute"
published: true
description: "Build a speculative decoding pipeline on Android using Vulkan compute shaders for draft models and NNAPI for verification, with adaptive batch scheduling."
tags: android, kotlin, architecture, performance
canonical_url: https://blog.mvpfactory.co/speculative-decoding-mobile-gpus-vulkan-compute
---

## What We Are Building

In this workshop, we are going to wire up a speculative decoding pipeline that runs entirely on-device on Android. A small ~150M parameter draft model will propose candidate tokens using Vulkan compute shaders, while a larger 3-7B verify model accepts or rejects them through NNAPI — all coordinated by a dynamic batch scheduler that adapts to thermal state and memory pressure.

The result: 2-3x lower per-token latency, pushing sub-200ms generation on flagship Android hardware. Let me show you how the pieces fit together.

## Prerequisites

- Android device with Vulkan 1.1+ compute support (2019 SoCs or newer)
- Android 12+ for `PowerManager.getThermalHeadroom()` API
- Familiarity with Kotlin and basic Vulkan concepts
- A quantized draft model (int4) and verify model (int8)

## Step 1: Understand the Split Architecture

Most teams get this wrong by running both models through the same accelerator. Split the pipeline instead.

| Component | Accelerator | Why |
|-----------|------------|-----|
| Draft model (~150M params) | Vulkan compute shaders | Direct GPU control, custom quantization kernels, no NNAPI overhead |
| Verify model (~3-7B params) | NNAPI (delegates to NPU/GPU) | Hardware-optimized int8/int4, vendor-tuned kernels |
| Batch scheduler | CPU | Lightweight coordinator, thermal/memory monitoring |
| KV-cache management | Shared GPU memory | Vulkan buffer exports via `VK_KHR_external_memory` |

A 7B model running autoregressively on a Snapdragon 8 Gen 3 generates roughly 8-12 tokens/second. With speculative decoding at K=5, server GPUs see 70-85% acceptance rates. The algorithm works. The engineering challenge is orchestrating two models across heterogeneous compute units without melting the phone.

## Step 2: Build the Vulkan Draft Pipeline

Here is the minimal setup to get this working. Custom GLSL compute shaders handle quantized matrix multiplications — 4-bit weights with fp16 accumulation hits the sweet spot for mobile GPU ALUs.

kotlin
class VulkanDraftModel(
private val device: VkDevice,
private val specDepth: Int = 5
) {
private val matmulPipeline: VkPipeline // int4 GEMV shader
private val kvCache: VkBuffer // exportable via external memory

fun proposeCandidates(inputTokenId: Int): IntArray {
    val candidates = IntArray(specDepth)
    var currentToken = inputTokenId

    for (i in 0 until specDepth) {
        bindDescriptorSets(currentToken, kvCache)
        vkCmdDispatch(commandBuffer, workgroupsX, 1, 1)
        candidates[i] = readArgmaxFromBuffer()
        currentToken = candidates[i]
    }
    return candidates
}

}


## Step 3: Wire Up the Adaptive Batch Scheduler

Here is a pattern I use in every project that involves on-device inference. You cannot run speculation depth K=8 when the device is thermal throttling at 45°C. The scheduler must adapt.

kotlin
class AdaptiveBatchScheduler(
private val thermalMonitor: ThermalMonitor,
private val memoryMonitor: GpuMemoryMonitor
) {
fun computeSpeculationDepth(): Int {
val thermalHeadroom = thermalMonitor.headroomFraction() // 0.0 - 1.0
val memoryAvailable = memoryMonitor.freeBufferMemoryMb()

    return when {
        thermalHeadroom < 0.15f -> 1  // near throttle: no speculation
        memoryAvailable < 64    -> 2  // memory-constrained
        thermalHeadroom < 0.40f -> 3  // warm but manageable
        else                    -> 6  // full speculation
    }
}

}


The scheduler polls `PowerManager.getThermalHeadroom()` on Android 12+ and reads `/sys/class/thermal/` zones as a fallback. GPU memory pressure comes from Vulkan's `vkGetPhysicalDeviceMemoryBudgetPropertiesEXT`.

On a Pixel 8 Pro, I measured the following thermal-adaptive behavior:

| Thermal State | Spec Depth | Tokens/sec | Acceptance Rate |
|---------------|-----------|------------|-----------------|
| Cool (<35°C) | 6 | 22-26 | 78% |
| Warm (35-42°C) | 3 | 16-19 | 74% |
| Hot (>42°C) | 1 | 9-11 | N/A (no speculation) |

## Step 4: Solve Zero-Copy KV-Cache Sharing

Both models need access to the key-value cache. The draft model builds speculative KV entries in Vulkan buffers. When the verify model accepts tokens, those entries become canonical. When it rejects, you roll back.

Use `VK_KHR_external_memory_fd` to export Vulkan buffers as file descriptors, then import them into NNAPI via `ANeuralNetworksMemory_createFromFd`. On a Snapdragon 8 Gen 3, a 512MB KV-cache copy costs ~8ms — that would erase most of your speculation benefit. In my benchmarks, this single zero-copy optimization was worth 15-20% throughput improvement.

## Gotchas

Here is the gotcha that will save you hours:

- **Pre-2019 SoCs** lack Vulkan 1.1 compute support entirely. The draft pipeline simply will not run. Check capabilities at startup and fall back gracefully.
- **NNAPI delegation is vendor-dependent.** Some NPU delegates reject model topologies silently. The docs do not mention this, but you will need logging at every delegation step to catch silent failures.
- **Memory budget is tighter than you think.** Devices with 6GB RAM leave roughly 1.5-2GB for both models after Android's runtime takes its share. You need aggressive quantization: int4 for the draft model, int8 for the verifier. There is no way around it.
- **Static speculation depth is a trap.** Build thermal-aware scheduling from day one. A fixed K will either waste thermals or leave performance on the table.

## Wrapping Up

The split-compute architecture — Vulkan for drafting, NNAPI for verification — is the only way to get parallel model execution on mobile. If you are doing on-device inference and have not explored this pattern yet, start with the Vulkan draft pipeline. It has the steepest learning curve, and everything else builds on top of it.

Build the scheduler early, invest in zero-copy KV-cache sharing, and respect the thermal envelope. That is how you get to 22+ tokens/second on a phone.

ARM NEON SIMD Intrinsics for Mobile Text Embedding: Building a Sub-10ms Semantic Search Pipeline That Runs Entirely On-Device

SoftwareDevs mvpfactory.io — Fri, 19 Jun 2026 07:08:09 +0000

---
title: "ARM NEON SIMD for Sub-10ms On-Device Semantic Search"
published: true
description: "A hands-on guide to replacing ONNX Runtime with hand-tuned ARM NEON SIMD kernels for int8 quantized matrix multiplication, hitting sub-10ms semantic search over 100K+ documents on mobile."
tags: android, ios, mobile, architecture
canonical_url: https://blog.mvpfactory.co/arm-neon-simd-for-sub-10ms-on-device-semantic-search
---

## What We Are Building

Let me show you how to drop from 36ms to under 7ms on a full semantic search pipeline — tokenization, embedding, and similarity scan — running entirely on a phone. No server round-trip.

We will replace ONNX Runtime with hand-tuned ARM NEON SIMD kernels for int8 quantized matrix multiplication, run E5-small (33M parameters, 384-dim output) on-device, and scan 100K+ document embeddings in under 3ms. By the end, you will understand the specific NEON intrinsics that do the heavy lifting and how to ship them cross-platform on Android and iOS.

## Prerequisites

- Familiarity with C and basic linear algebra (matrix multiply, dot product)
- An ARMv8 target device (any phone shipped since ~2019)
- NDK setup for Android or Xcode with C bridging headers for iOS
- A pre-quantized int8 embedding model (E5-small works well)

## Step 1: Understand the Pipeline

Here is the minimal architecture. Three stages, each with a latency target:

| Stage | Operation | Target Latency |
|-------|-----------|----------------|
| Tokenization | BPE tokenize query string | < 1ms |
| Embedding | Int8 quantized forward pass via NEON GEMM | < 6ms |
| Search | Vectorized dot-product over 100K embeddings | < 3ms |

The key decision: bypass the inference runtime entirely for the embedding step. Generic runtimes like ONNX Runtime carry operator dispatch overhead, suboptimal memory allocation patterns, and operator fusion gaps. We write NEON-native GEMM kernels that operate directly on pre-quantized int8 weights.

## Step 2: Write the NEON GEMM Kernel

ARM NEON gives you 128-bit SIMD registers, processing 16 int8 values simultaneously. Here is the core kernel:

c
void neon_gemm_int8(const int8_t* A, const int8_t* B,
int32_t* C, int M, int N, int K) {
for (int i = 0; i < M; i++) {
for (int j = 0; j < N; j += 4) {
int32x4_t acc = vdupq_n_s32(0);
for (int k = 0; k < K; k += 16) {
int8x16_t a_vec = vld1q_s8(&A[i * K + k]);
int8x16_t b_vec = vld1q_s8(&B[j * K + k]);
int16x8_t prod_lo = vmull_s8(vget_low_s8(a_vec),
vget_low_s8(b_vec));
int16x8_t prod_hi = vmull_s8(vget_high_s8(a_vec),
vget_high_s8(b_vec));
acc = vpadalq_s16(acc, prod_lo);
acc = vpadalq_s16(acc, prod_hi);
}
vst1q_s32(&C[i * N + j], acc);
}
}
}


On ARMv8.2+ devices, you get `vdotq_s32` — a fused dot-product instruction that processes 4 int8 multiplies and accumulates in a single cycle. This single intrinsic is the difference between "workable" and "instant":

c
int32x4_t acc = vdupq_n_s32(0);
acc = vdotq_s32(acc, a_vec, b_vec); // 4x throughput improvement


## Step 3: Vectorize the Similarity Search

Once you have a 384-dim query embedding, scanning 100K document embeddings is a vectorized dot-product problem:

c
float neon_dot_f32(const float* a, const float* b, int dim) {
float32x4_t sum = vdupq_n_f32(0.0f);
for (int i = 0; i < dim; i += 4) {
float32x4_t va = vld1q_f32(&a[i]);
float32x4_t vb = vld1q_f32(&b[i]);
sum = vfmaq_f32(sum, va, vb);
}
return vaddvq_f32(sum);
}


For 100K documents at 384 dimensions, that is ~38.4M multiply-adds. NEON processes 4 per cycle, and at 2.5 GHz on a typical big core, we consistently land under 3ms thanks to L1 cache locality on sequential scans.

## Step 4: Ship Cross-Platform

The same NEON intrinsics compile directly via Clang on iOS since Apple Silicon shares the ARMv8 ISA. Wrap your kernels in a C library, expose via JNI on Android and a C bridging header on iOS. If you are using Kotlin Multiplatform for your application layer, this native SIMD layer sits cleanly beneath your shared Kotlin search API.

## The Numbers

Measured on Snapdragon 8 Gen 2 (Cortex-X3 big core), E5-small:

| Metric | ONNX Runtime (fp32) | ONNX Runtime (int8) | Hand-tuned NEON (int8) |
|--------|---------------------|---------------------|------------------------|
| Embedding latency | 28ms | 14ms | 4.7ms |
| 100K similarity search | 8ms | 8ms | 2.1ms |
| Total pipeline | 36ms | 22ms | 6.8ms |
| Peak memory | 142MB | 89MB | 61MB |
| APK size overhead | +8MB (runtime) | +8MB | +0.2MB (kernel lib) |

3x faster than quantized ONNX Runtime, 5x faster than fp32, with less than half the memory and virtually zero binary size overhead.

## Gotchas

- **Always provide a fallback path.** Not every device supports ARMv8.2+ dot-product instructions. Use `getauxval(AT_HWCAP)` on Android for runtime feature detection, or compile-time targeting on iOS. Ship both the `vdotq_s32` path and the widening multiply-accumulate path.
- **Memory-map your index.** Store your 100K document embeddings as a flat `mmap`ed binary file. The docs do not mention this, but skipping deserialization and letting the NEON scan operate directly on mapped memory with zero copy is where you reclaim the last couple of milliseconds.
- **Watch your K dimension alignment.** The inner loop steps by 16 (`k += 16`). If your model dimension is not a multiple of 16, you need padding or a scalar tail loop. Forgetting this is a silent correctness bug — you will get wrong results, not a crash.
- **Do not assume little-core performance.** All benchmarks above use the big core. Background tasks on efficiency cores will be 2-3x slower. Pin your search thread to big cores via `sched_setaffinity` on Android.

## Wrapping Up

Here is the minimal setup to get this working: quantize your model to int8, write NEON GEMM kernels directly, target `vdotq_s32` with a fallback, and memory-map your document index. The general-purpose runtime overhead is real and measurable. For latency-sensitive paths on mobile, bypass it.

Quantized LoRA Adapters for On-Device LLMs: Hot-Swapping Task-Specific Behaviors on Android Without Reloading the Base Model

SoftwareDevs mvpfactory.io — Thu, 18 Jun 2026 14:05:02 +0000

---
title: "QLoRA Adapters on Android: Hot-Swap LLM Tasks in Under 100ms"
published: true
description: "Load a 4-bit quantized base model once on Android and hot-swap 2MB LoRA adapters for different tasks using llama.cpp, Kotlin, and NEON optimizations."
tags: kotlin, android, architecture, mobile
canonical_url: https://blog.mvpfactory.co/qlora-adapters-android-hot-swap-llm-tasks
---

## What We Will Build

Let me show you a pattern I use in every project that runs on-device LLMs: loading a single 4-bit quantized base model into memory via `mmap`, then dynamically swapping ~2MB LoRA adapter weights to switch between summarization, code review, translation — any task you need. All in under 100ms on modern Android hardware.

By the end of this tutorial, you will have a lifecycle-aware Kotlin service that manages a base model and multiple QLoRA adapters, with proper native memory cleanup and an LRU cache for instant task switching.

## Prerequisites

- Android Studio with NDK installed
- A device or emulator with at least 8GB RAM (Pixel 7+ recommended)
- `llama.cpp` built for Android with LoRA support enabled
- A GGUF-quantized base model (Q4_K_M, 7B parameters)
- One or more LoRA adapter files (~1.5–3MB each)
- Familiarity with Kotlin coroutines and Jetpack Lifecycle

## Step 1: Understand Why One Model Per Task Does Not Scale

Most teams start with the obvious approach — one fine-tuned model per task. Here is the gotcha that will save you hours of frustration: a 7B parameter model quantized to 4-bit (Q4_K_M) runs around 3.8–4.2GB in RAM. Need three tasks? That is 12GB of model weight, untenable on any shipping Android device.

The mistake is treating model specialization as a model-level concern when it is actually a **weight-delta concern**. QLoRA adapters encode task-specific behavior as small rank-decomposition matrices layered on top of a frozen base model.

| Approach | RAM for 3 tasks | Cold-start latency | Task-switch latency |
|---|---|---|---|
| 3 separate Q4 models | ~12.0 GB | 8–12s each | 8–12s (full reload) |
| 1 base + 3 LoRA adapters | ~4.2 GB + 6 MB | 8–12s (once) | 50–90ms |
| 1 merged model per task | ~12.0 GB on disk | 8–12s each | 8–12s (full reload) |

The adapter approach cuts both memory and switching latency by orders of magnitude.

## Step 2: The mmap Trick That Makes Sub-100ms Swaps Possible

The docs do not mention this, but the key to fast adapter swaps is how `llama.cpp` handles model loading on Android. When you load a GGUF model with `mmap` enabled, the OS maps the file directly into virtual address space without copying it into the process heap. Base model weights get page-faulted on demand from flash storage.

LoRA adapters, by contrast, are small enough to live entirely in resident memory. A swap means:

1. Deallocating the current adapter's rank-decomposition matrices (~2MB)
2. Allocating and loading the new adapter (~2MB)
3. **No base model teardown or reload**

On a Pixel 8 with UFS 4.0 storage, this benchmarks consistently at 50–90ms. The base model's memory-mapped pages stay warm in the page cache across swaps.

## Step 3: NEON-Optimized Matrix Fusion for Merged Inference

You do not want to compute `base_output + lora_output` as two separate matrix multiplications at inference time. The better path is fusing the LoRA weights into the base weights for active layers using ARM NEON intrinsics.

The math: for a given layer, the effective weight becomes `W_eff = W_base + (alpha/r) * B * A`, where `A` and `B` are the low-rank matrices and `r` is the adapter rank. With rank 8–16 (typical for mobile adapters), this fusion takes 15–30ms across all target layers on an 8-core ARM processor using NEON SIMD.

Your actual inference path sees **zero overhead** from using an adapter versus a natively fine-tuned model. That is the whole point.

## Step 4: Build the Kotlin Service with Lifecycle-Aware Adapter Management

Here is the minimal setup to get this working. The lifecycle management is where mobile teams stumble — the model loading and adapter math are well-documented, but keeping native memory from leaking when Android kills your activity is not.

kotlin
class AdapterManager(
private val baseModel: LlamaModel
) : DefaultLifecycleObserver {

private var activeAdapter: LoraAdapter? = null
private val adapterCache = LruCache<String, ByteArray>(3) // cache top 3

suspend fun switchAdapter(taskId: String): Result<Long> {
    val startNs = System.nanoTime()
    activeAdapter?.detach()

    val weights = adapterCache.get(taskId)
        ?: loadAdapterFromAssets(taskId).also { adapterCache.put(taskId, it) }

    activeAdapter = baseModel.attachLoraAdapter(weights)
    val elapsedMs = (System.nanoTime() - startNs) / 1_000_000
    return Result.success(elapsedMs)
}

override fun onStop(owner: LifecycleOwner) {
    activeAdapter?.detach()
    activeAdapter = null
}

}


Key design decisions:

- The `LruCache` holds adapter bytes for up to 3 adapters. At ~2MB each, the 6MB cache cost is negligible, and cache hits eliminate file-read latency.
- Detaching adapters in `onStop` prevents leaked native memory when the app backgrounds. This matters because `llama.cpp` allocations live outside the JVM heap — the garbage collector will never touch them.
- The `suspend` function keeps the swap off the main thread while remaining trivially callable from ViewModels.

This maps well to on-device agentic workflows. An on-device agent can break a goal into steps — one step might need an intent-analysis adapter, the next a response-generation adapter, and a third a summarization adapter. Sub-100ms swaps make multi-adapter pipelines viable on mobile.

## Step 5: Know Your Memory Budget

| Component | RAM (resident) | RAM (virtual/mapped) |
|---|---|---|
| Base model (Q4_K_M, 7B) | ~800 MB active pages | 4.0 GB mapped |
| Active LoRA adapter | 2 MB | 2 MB |
| Cached adapters (x2) | 4 MB | 4 MB |
| Fusion workspace (NEON) | 12 MB | 12 MB |
| **Total** | **~818 MB** | **~4.02 GB** |

The distinction between resident and mapped memory matters. Android's `mmap` means your app's PSS (Proportional Set Size) reflects only actively accessed pages, not the full model file. Most OEMs' low-memory-killer thresholds will not trigger against ~800MB resident on flagships with 8–12GB RAM.

## Gotchas

- **Native memory leaks are silent killers.** `llama.cpp` allocations live outside the JVM heap. If you forget to detach adapters in `onStop`, your app will crash after extended sessions. I have seen this happen to teams repeatedly. Use `DefaultLifecycleObserver` — do not rely on `onDestroy`.
- **Fuse at swap time, not at inference time.** If you compute `W_base + LoRA_delta` per token, you add latency to every single generation step. Pay the 15–30ms fusion cost once during the swap and get native performance on every token after.
- **Do not skip the LRU cache.** Reading 2MB from assets on every swap adds unnecessary I/O. Cache your top adapters in memory — the 6MB cost is trivial compared to the base model.
- **Watch your adapter rank.** Rank 8–16 is the sweet spot for mobile. Higher ranks give marginal quality gains but increase fusion time and adapter file size significantly.
- **Test on real hardware.** Emulator benchmarks are meaningless for `mmap` and NEON performance. Always profile on a physical device with UFS storage.

During long benchmarking sessions like these, I keep [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) running for break reminders and guided desk exercises — no amount of elegant adapter architecture helps if you are wrecked from six hours of unbroken profiling.

## Conclusion

Load your base model once with `mmap`, then treat adapters as the unit of task specialization. The per-adapter cost (~2MB, ~70ms swap) makes multi-task on-device LLMs practical today on flagship Android hardware. Fuse LoRA weights into base weights using NEON SIMD before inference, and bind adapter lifecycle to Android component lifecycle to prevent the silent native memory leaks that crash apps after extended sessions.

The pattern is simple: one base model, many tiny adapters, lifecycle-aware cleanup. That is your path to shipping multi-task LLMs on Android without melting the device.

CRDTs in Kotlin Multiplatform: Kill Your Sync Server

SoftwareDevs mvpfactory.io — Thu, 18 Jun 2026 07:35:18 +0000

What we're building

By the end of this tutorial, you'll have a working mental model — and working code — for implementing Conflict-Free Replicated Data Types in Kotlin Multiplatform shared code. We'll build an LWW-Register, compare state-based vs operation-based sync strategies, and walk through the architecture that lets you replace your entire sync backend with dumb blob storage.

Let me show you a pattern I use in every project with offline-first requirements.

Prerequisites

Kotlin Multiplatform project set up with commonMain, Android, and iOS targets
Familiarity with Kotlin data classes and basic merge logic
SQLDelight (recommended for local persistence)

Step 1: Understand why custom sync servers fail

Most teams build a centralized arbiter — a server that receives conflicting writes, applies "last write wins" at the field level, and hopes for the best. This creates a single point of failure and an ever-growing surface area of edge-case conflict logic nobody wants to maintain.

CRDTs flip the model. Every replica converges to the same state given the same set of updates. Mathematically guaranteed, no coordination required.

Step 2: Pick the right primitives

Not every CRDT is practical on constrained devices. Here is the minimal setup to get this working:

Primitive	Use case	Merge cost	Payload overhead
LWW-Register	User profile fields, settings	O(1)	Minimal: timestamp + value
G-Counter	Analytics events, view counts	O(n) where n = replicas	Grows linearly with replica count
PN-Counter	Inventory, cart quantities	O(n)	2x G-Counter
RGA	Collaborative text, ordered lists	O(log n) amortized	Tombstones accumulate over time
OR-Set	Tags, favorites, selections	O(n) per element	Causal metadata per item

For most mobile apps, LWW-Registers and OR-Sets cover the majority of sync needs. RGA is only necessary when you need ordered, collaborative sequences.

Step 3: Implement an LWW-Register in `commonMain`

One implementation, every platform. Drop this into your shared module:

data class LWWRegister<T>(
    val value: T,
    val timestamp: Long,
    val nodeId: String
) {
    fun merge(other: LWWRegister<T>): LWWRegister<T> = when {
        other.timestamp > this.timestamp -> other
        other.timestamp < this.timestamp -> this
        else -> if (other.nodeId > this.nodeId) other else this
    }
}

The nodeId tiebreaker matters more than people realize. Without it, identical timestamps produce non-deterministic merges, which violates the convergence guarantee. The docs do not mention this, but most tutorials skip this detail entirely.

Step 4: Choose state-based vs operation-based

Dimension	State-based (CvRDT)	Operation-based (CmRDT)
Network requirement	Unreliable (idempotent merge)	Exactly-once delivery needed
Payload size	Full state on each sync	Individual operations
Infrastructure complexity	Lower: just exchange states	Higher: needs causal ordering layer
Bandwidth on constrained networks	Higher per message	Lower per message
Implementation difficulty	Simpler merge functions	Requires operation log + delivery guarantees

On mobile, state-based CRDTs are the pragmatic default. 3G connections drop mid-sync. Apps get backgrounded and sockets die. Requiring exactly-once delivery for operation-based CRDTs means building a reliable causal broadcast layer — which reintroduces the backend complexity you were trying to escape.

If your documents grow large, delta-state CRDTs offer a hybrid approach: transmit only the state diff since last sync, reclaiming bandwidth without sacrificing idempotency.

Step 5: Handle vector clocks without fear

Vector clocks track causal ordering across replicas. Here is the gotcha that will save you hours: on mobile, the constraint is friendlier than it sounds. Most users have a bounded number of devices. A vector clock with entries for a phone, tablet, and laptop is three integers. That's nothing.

Prune entries for devices inactive beyond a threshold, and the metadata stays compact. Store vector clocks alongside each CRDT in SQLDelight, and compare them during sync to detect concurrent updates versus causal successors.

For Automerge integration, wrap the native Rust-based library via expect/actual declarations — JNI on Android, C interop on iOS through the shared boundary.

Step 6: Delete your sync service

┌──────────┐     ┌──────────┐     ┌──────────┐
│ Android  │     │   iOS    │     │ Desktop  │
└────┬─────┘     └────┬─────┘     └────┬─────┘
     │                │                │
     │   CRDT State Blobs (opaque)     │
     └────────┬───────┴───────┬────────┘
              ▼               ▼
        ┌───────────────────────┐
        │   Dumb Storage Relay  │
        │  (S3 / Cloud Storage) │
        │  No conflict logic    │
        └───────────────────────┘

Your "backend" becomes a storage relay. It holds opaque CRDT state blobs, knows nothing about your data model, resolves zero conflicts, and scales with commodity object storage pricing. You delete the sync service, its tests, its deployment pipeline, and its on-call rotation.

Gotchas

Missing nodeId tiebreaker: Without deterministic tiebreaking on equal timestamps, your LWW-Register violates convergence. Always include it.
Reaching for operation-based CRDTs too early: They require exactly-once delivery guarantees. On mobile networks, that means building infrastructure you were trying to avoid.
Optimizing payload size before measuring: State-based payloads for typical mobile data sets (preferences, local lists, document fragments) stay well within acceptable bounds. Reach for delta-state variants only when payload sizes become a measured problem, not a theoretical one.
Overestimating vector clock overhead: You're not running thousands of nodes. You're tracking three devices.

Conclusion

Start with LWW-Registers and OR-Sets in commonMain. These two primitives cover user settings, favorites, tags, and most entity-level sync needs. Write platform-agnostic property tests that verify convergence. Default to state-based CRDTs — the idempotent merge model tolerates unreliable networks without additional infrastructure.

Once your clients can independently merge state, your backend doesn't need conflict resolution logic anymore. Reduce it to authenticated blob storage and spend that engineering time on product work instead.

Redis Streams as Your Startup's Event Bus

SoftwareDevs mvpfactory.io — Wed, 17 Jun 2026 14:15:40 +0000

---
title: "Redis Streams as Your Startup's Event Bus: A Ktor Workshop"
published: true
description: "Build a production event bus with Redis Streams, consumer groups, and Ktor coroutines — delaying your Kafka migration until you actually need it."
tags: kotlin, architecture, api, backend
canonical_url: https://blog.mvpfactory.co/redis-streams-as-your-startups-event-bus
---

## What We're Building

In this workshop, I'll walk you through building a production event bus using Redis Streams with Ktor and Exposed. By the end, you'll have a working order-processing consumer with exactly-once semantics, automatic dead-letter routing, and webhook fan-out — all without touching Kafka.

Redis Streams, added in Redis 5.0, give you append-only log semantics with consumer groups. Those are the exact primitives you need for an event bus, and you're almost certainly already running Redis. Let me show you a pattern I use in every project.

## Prerequisites

- Kotlin + Ktor project with coroutines
- Exposed ORM configured with your database
- Redis 6.2+ (for `XAUTOCLAIM`)
- Lettuce Redis client with coroutine support

## Step 1: Understand the Consumer Group Mental Model

Before writing code, here is the mapping that makes everything click:

| Concept | Kafka | Redis Streams |
|---|---|---|
| Log append | `producer.send()` | `XADD` |
| Consumer group read | `poll()` | `XREADGROUP` |
| Offset commit | `commitSync()` | `XACK` |
| Uncommitted offsets | Consumer lag | Pending Entry List (PEL) |
| Rebalancing | Automatic partition reassignment | Manual via `XCLAIM` / `XAUTOCLAIM` |
| Retention | Time/size-based | `MAXLEN` / `MINID` |

The key difference: Redis Streams gives you per-message acknowledgment out of the box. No batching offset commits, no rebalancing storms.

## Step 2: Build the Coroutine Consumer

Here is the minimal setup to get this working. This consumer reads from a group with exactly-once semantics via idempotency keys:

kotlin
class StreamConsumer(
private val redis: RedisCoroutinesCommands,
private val db: Database,
private val stream: String,
private val group: String,
private val consumerId: String
) {
suspend fun consume() = coroutineScope {
try { redis.xgroupCreate(
XReadArgs.StreamOffset.from(stream, "0-0"), group,
XGroupCreateArgs.Builder.mkstream()
) } catch (_: Exception) { /* group exists */ }

    while (isActive) {
        val messages = redis.xreadgroup(
            Consumer.from(group, consumerId),
            XReadArgs.Builder.count(50).block(Duration.ofSeconds(2)),
            XReadArgs.StreamOffset.lastConsumed(stream)
        )
        messages?.forEach { msg ->
            processWithIdempotency(msg)
        }
    }
}

private suspend fun processWithIdempotency(msg: StreamMessage<String, String>) {
    val idempotencyKey = msg.id
    transaction(db) {
        val exists = ProcessedEvents
            .select { ProcessedEvents.eventId eq idempotencyKey }
            .count() > 0
        if (!exists) {
            ProcessedEvents.insert { it[eventId] = idempotencyKey }
            handleEvent(msg.body)
        }
    }
    redis.xack(stream, group, msg.id)  // ACK only after DB commit
}

}


The ordering here matters: DB commit happens *before* `XACK`. If the process crashes between commit and ACK, the message stays in the PEL and gets redelivered, but the idempotency check prevents double processing. This is exactly-once *effective* processing.

## Step 3: Add Backpressure with XPENDING and Dead-Letter Routing

The Pending Entry List is your backpressure signal. This monitor coroutine detects stuck consumers and reclaims their messages:

kotlin
launch {
while (isActive) {
delay(30_000)
val stale = redis.xautoclaim(
stream, XAutoClaimArgs.Builder
.xautoclaim(group, consumerId, 60_000, "0-0")
)
stale.messages.forEach { msg ->
val deliveryCount = redis.xpending(stream, group, msg.id, msg.id, 1)
.firstOrNull()?.nacks ?: 0
if (deliveryCount > 3) {
redis.xadd("${stream}:dlq", msg.body)
redis.xack(stream, group, msg.id)
}
}
}
}


| Signal | Action |
|---|---|
| PEL size > threshold | Pause new reads via `XPENDING` summary check |
| Message idle > 60s | Reclaim with `XAUTOCLAIM` |
| Delivery count > 3 | Route to dead-letter queue |
| Consumer lag growing | Add consumer instances to the group |

## Step 4: Webhook Fan-Out with Multiple Consumer Groups

For webhook fan-out, use multiple consumer groups on the same stream. Each group gets every message independently:

kotlin
val groups = listOf("webhook-delivery", "analytics-pipeline", "notification-sender")
groups.forEach { group ->
launch { StreamConsumer(redis, db, "orders", group, "$group-${hostId}").consume() }
}


Each group maintains its own PEL and offset. Zero coordination between them.

## Gotchas

Here is the gotcha that will save you hours:

- **ACK after commit, always.** If you `XACK` before your DB transaction commits and the process crashes, you lose the event silently. The PEL exists to protect you — let it.
- **`XAUTOCLAIM` requires Redis 6.2+.** On older versions you need manual `XCLAIM` loops. Check your version before deploying.
- **RAM is your retention ceiling.** Redis Streams live in memory. You get hours to days of retention, not weeks. Use `MAXLEN` or `MINID` trimming aggressively.
- **The docs do not mention this, but** the crossover point to Kafka is concrete: sustained throughput above ~80K msg/s, retention needs beyond 48 hours, or more than ~20 independent consumer groups. A single Redis node sustains ~120K msg/s sustained (~200K burst) at <2ms p99 — but storage efficiency is low compared to Kafka's disk-based approach.

## Conclusion

You can build a production event bus in an afternoon with Redis Streams instead of a quarter-long Kafka rollout. Wire up `XREADGROUP` + `XACK`, add idempotency keys in your Ktor consumers, and monitor backpressure through `XPENDING`. Don't plan a Kafka migration *timeline* — plan a Kafka migration *trigger* with concrete thresholds. Migrate when you hit them. Not before.

PostgreSQL Advisory Locks for Distributed Job Scheduling

SoftwareDevs mvpfactory.io — Wed, 17 Jun 2026 08:51:43 +0000

---
title: "PostgreSQL Advisory Locks for Distributed Job Scheduling"
published: true
description: "Replace Redis and SQS with PostgreSQL advisory locks for distributed job scheduling. Benchmarks, lock strategies, and PgBouncer gotchas for 10K jobs/minute."
tags: postgresql, architecture, devops, cloud
canonical_url: https://blog.mvpfactory.co/postgresql-advisory-locks-for-distributed-job-scheduling
---

## What We Will Build

Let me show you a pattern I use in every project that needs distributed job scheduling: PostgreSQL advisory locks. By the end of this tutorial, you will have a production-ready job claiming system using `pg_try_advisory_xact_lock` and `FOR UPDATE SKIP LOCKED` — no Redis, no SQS, no new infrastructure.

In my benchmarks, this handles 10K jobs/minute on a single `db.r6g.xlarge` with sub-millisecond lock acquisition, outperforming Redis `SETNX` in latency when your workers already hold database connections.

## Prerequisites

- PostgreSQL 9.5+ (for `SKIP LOCKED`)
- Basic understanding of SQL transactions and connection pooling
- A multi-instance deployment where workers compete for jobs

## Step 1: Understand Advisory Locks

Advisory locks are database-level cooperative locks that don't attach to any table or row. They live in shared memory. Cheap to acquire, cheap to release.

Here is the minimal setup to get this working:

sql
-- Attempt to claim job 42 within current transaction
SELECT pg_try_advisory_xact_lock(42);
-- Returns true if acquired, false if another session holds it


The `_try_` variant is non-blocking. That matters for job schedulers where you want workers to skip contested jobs, not queue behind them.

## Step 2: Choose Transactional vs. Session Locks

This is the decision that will make or break your implementation.

| Property | `pg_try_advisory_xact_lock` | `pg_try_advisory_lock` |
|---|---|---|
| Release | Auto on `COMMIT`/`ROLLBACK` | Manual `pg_advisory_unlock()` |
| Leak risk | None | High if app crashes |
| PgBouncer safe | Yes (transaction mode) | No (requires session mode) |
| Use case | Job claiming, idempotent ops | Leader election, long tasks |

Use transactional locks unless you explicitly control your connection pool at the session level. Seriously. I will explain why in the Gotchas section.

## Step 3: Use Composite Lock Keys with Namespacing

Advisory locks accept either a single `bigint` or two `integer` arguments. The two-integer form gives you natural namespacing:

sql
-- (namespace, job_id)
SELECT pg_try_advisory_xact_lock(1, job_id); -- namespace 1 = email jobs
SELECT pg_try_advisory_xact_lock(2, job_id); -- namespace 2 = webhook jobs


When you need to lock on a text key, hash it to `bigint`:

sql
SELECT pg_try_advisory_xact_lock(hashtextextended('order:' || order_id, 0));


The birthday paradox governs collision probability. With `bigint` (2^63 positive values):

| Distinct lock IDs | Collision probability |
|---|---|
| 10,000 | ~0.000000005 |
| 100,000 | ~0.0000005 |
| 1,000,000 | ~0.00005 |
| 10,000,000 | ~0.005 (0.5%) |

Below 1M concurrent distinct lock IDs, collisions are negligible. Past that, use the two-integer form with explicit namespace separation.

## Step 4: Implement the Job Claiming Pattern

Here is the core pattern I deploy in multi-instance schedulers:

sql
WITH claimable AS (
SELECT id FROM jobs
WHERE status = 'pending'
AND scheduled_at <= now()
ORDER BY priority DESC, scheduled_at ASC
LIMIT 50
FOR UPDATE SKIP LOCKED
)
UPDATE jobs SET status = 'processing', worker_id = $1, claimed_at = now()
FROM claimable WHERE jobs.id = claimable.id
RETURNING jobs.*;


`FOR UPDATE SKIP LOCKED` (PostgreSQL 9.5+) does the heavy lifting for row-level contention. Advisory locks layer on top for distributed coordination: preventing duplicate scheduling across cron triggers, or ensuring singleton execution of workflows that must not run concurrently.

## Step 5: Monitor Lock Contention

Production visibility comes from `pg_stat_activity` joined with `pg_locks`:

sql
SELECT pid, mode, granted, objid,
now() - query_start AS lock_duration,
query
FROM pg_locks l
JOIN pg_stat_activity a ON l.pid = a.pid
WHERE locktype = 'advisory'
AND NOT granted
ORDER BY lock_duration DESC;


Alert when ungranted advisory lock waits exceed your SLA. In healthy systems running 10K jobs/minute, I see fewer than 5 contested locks at any given instant using the `SKIP LOCKED` pattern above.

## Benchmarks: Where Advisory Locks Win and Where They Don't

Tested on AWS. PostgreSQL 15 on `db.r6g.xlarge`, Redis 7 on `cache.r6g.large`, 8 worker instances on `c6g.medium`:

| Metric | PG advisory locks | Redis SETNX |
|---|---|---|
| Lock acquire p50 | 0.3 ms | 0.4 ms |
| Lock acquire p99 | 1.2 ms | 0.8 ms |
| Throughput ceiling | ~12K jobs/min | ~50K+ jobs/min |
| Failure mode | Connection exhaustion | Memory exhaustion |
| Added infra | None | Redis cluster + sentinel |

Below ~10K jobs/minute, advisory locks match or beat Redis on latency because you eliminate the network hop to a separate service. Past that threshold, PostgreSQL's lock manager and connection limits become the bottleneck. Redis scales horizontally with less friction.

The docs do not mention this, but this is the honest tradeoff. Advisory locks win on simplicity and operational cost. Redis wins on raw throughput ceiling. Most teams I have worked with are well below the 10K/min line and don't need Redis for this.

## Gotchas

Here is the gotcha that will save you hours.

**PgBouncer in transaction mode silently breaks session locks.** Most teams get this wrong the same way: they reach for session-level locks for leader election, then deploy behind PgBouncer in transaction mode. PgBouncer reassigns the underlying connection between transactions, so your session lock now belongs to a different application session. The lock is silently orphaned. I have debugged this in production more than once.

If you run PgBouncer, configure a separate pool for your job workers:

ini
[databases]
app = host=pg-primary dbname=myapp pool_mode=transaction
workers = host=pg-primary dbname=myapp pool_mode=session pool_size=20


Your application traffic gets the efficiency of transaction pooling. Your job workers get stable sessions for advisory lock correctness. Don't mix them.

**Hash collisions past 100K distinct lock IDs.** When hashing text keys to `bigint`, collisions become a real problem past 100K distinct lock IDs. Use the two-integer composite form with explicit namespace separation at scale.

**Mixing connection pools is the #1 production failure.** Mixing transactional web traffic and long-running job connections in the same pool is the single most common production failure I see with this pattern. Every time. Isolate your worker connection pool.

## Conclusion

Start with `pg_try_advisory_xact_lock` and `SKIP LOCKED`. This combination covers the vast majority of distributed job scheduling needs without adding infrastructure, and transactional locks cannot leak or break under PgBouncer. Benchmark when you approach 10K jobs/minute — let your actual numbers decide, not architectural assumptions.

Profile-Guided Optimization for Android App Startup

SoftwareDevs mvpfactory.io — Tue, 16 Jun 2026 14:29:59 +0000

---
title: "Profile-Guided Optimization: Android Cold Start Under 400ms"
published: true
description: "A hands-on walkthrough of Baseline Profiles, Cloud Profiles, and DEX layout reordering — the pipeline that cut our cold start from 1.2s to 380ms."
tags: android, kotlin, mobile, performance
canonical_url: https://blog.mvpfactory.co/profile-guided-optimization-android-cold-start-under-400ms
---

## What We're Building

By the end of this tutorial, you'll have a complete profile-guided optimization pipeline that seeds ART's ahead-of-time compiler at install time, reorders your DEX layout for minimal page faults, and validates everything with Macrobenchmark. We took a 1.2s cold start down to 380ms with this exact setup. Let me show you each layer.

## Prerequisites

- Android Gradle Plugin 7.3+ (R8 enabled by default)
- Macrobenchmark library (`androidx.benchmark:benchmark-macro-junit4`)
- A physical device or managed device for profiling (emulators skew results)
- Familiarity with Perfetto traces (helpful, not required)

## Step 1: Understand What ART Does Without You

ART runs in multiple compilation modes, and the one your users hit on first launch is the worst one:

| Mode | When It Runs | Startup Impact |
|------|-------------|----------------|
| Interpret-only | First install, no profile | Slowest: bytecode interpreted at runtime |
| Speed-profile | After profile collection (idle maintenance) | Fast for profiled methods, slow for the rest |
| Speed | Full AOT (rare, OEM-triggered) | Fastest but largest on-disk footprint |

On a fresh install with no Baseline Profile, ART defaults to interpret-only. The JIT compiles hot methods at runtime, writes a profile to disk, and only during idle device maintenance does `bg-dexopt` AOT-compile your critical path. Your user's first session — the one that determines retention — runs on the slowest mode. That's the gap we fill.

## Step 2: Generate Baseline Profiles in CI

A Baseline Profile lists classes and methods to AOT-compile at install time. Here is the minimal setup to get this working with Macrobenchmark:

kotlin
@ExperimentalBaselineProfilesApi
@RunWith(AndroidJUnit4::class)
class StartupProfile {
@get:Rule
val rule = BaselineProfileRule()

@Test
fun generateBaselineProfile() = rule.collect(
    packageName = "com.example.app",
    maxIterations = 5,
    stableIterations = 3
) {
    pressHome()
    startActivityAndWait()
    device.findObject(By.res("main_feed"))
        .wait(Until.hasObject(By.res("feed_item")), 5000)
}

}


Our generated profile covered about 12% of total DEX methods, but those methods represented 94% of wall-clock time in the first 500ms of startup. Wire this into your release pipeline — not a one-time task.

## Step 3: Enable DEX Layout Reordering

Here is the gotcha that will save you hours. Baseline Profiles alone got us from 1,204ms to 620ms. Adding DEX startup layout reordering pushed us to 445ms — a bigger incremental gain than the profile itself. It is a one-line Gradle property:

properties
android.enableStartupDex=true


This tells R8 to reorder classes in `classes.dex` so startup-critical classes are physically contiguous. It matters because of page faults — a 4KB memory page loaded from disk contains multiple class definitions. Scattered startup classes mean the kernel loads pages full of irrelevant data. Reordering cut our page faults by 30–50%.

## Step 4: Let Cloud Profiles Compound

Starting with API 28+, Google Play aggregates anonymized runtime profiles and delivers them to new installs. Combined with the previous steps, here are the real numbers:

| Configuration | Page Faults (Median) | Cold Start (P50) | Cold Start (P95) |
|--|--|--|--|
| No profile, default layout | 1,847 | 1,204ms | 1,890ms |
| Baseline Profile only | 1,210 | 620ms | 980ms |
| Baseline + DEX reorder | 780 | 445ms | 710ms |
| Baseline + DEX reorder + Cloud | 690 | 380ms | 590ms |

## Step 5: Validate With Macrobenchmark

The docs do not mention this, but `CompilationMode.Full` will make your numbers look great and tell you nothing useful. Use `Partial` to simulate real-world conditions:

kotlin
@get:Rule
val benchmarkRule = MacrobenchmarkRule()

@test
fun startupCold() = benchmarkRule.measureRepeated(
packageName = "com.example.app",
metrics = listOf(StartupTimingMetric()),
iterations = 10,
startupMode = StartupMode.COLD,
compilationMode = CompilationMode.Partial(
baselineProfileMode = BaselineProfileMode.Require
)
) {
startActivityAndWait()
}


Benchmark across API level brackets: 24–27 (no Cloud Profile support), 28–30 (Cloud Profiles, older ART), and 31–35 (latest ART with improved compilation). A profile that cuts cold start by 60% on API 33 may only yield 30% on API 26.

## Gotchas

- **Stale profiles kill gains silently.** Generate profiles in CI on every release. I've seen teams lose 200ms over a few months from profile drift alone.
- **R8 invalidates your profile.** Method inlining causes inlined methods to disappear from DEX. Always regenerate profiles post-R8.
- **Cloud Profiles lag 24–48 hours.** Early adopters — your most engaged users — get no benefit. Baseline Profiles give you deterministic, day-zero coverage. Relying solely on Cloud Profiles is a mistake.
- **Watch the right Perfetto trace points:** `bindApplication` (framework init), `activityStart` (your `onCreate`), and `reportFullyDrawn` (your declared "ready" signal).

## Wrapping Up

Let me show you a pattern I use in every project: Baseline Profile generation in CI, DEX reorder enabled, Macrobenchmark gating the release across API brackets. Each layer compounds on the last. Skip any one of them and you leave real performance on the table. Start with Step 2, measure the delta, then stack each layer and watch the numbers drop.

KV Cache Quantization for On-Device LLMs

SoftwareDevs mvpfactory.io — Tue, 16 Jun 2026 09:04:51 +0000

---
title: "KV Cache Quantization: Fitting Llama 3.2 3B in 2 GB RAM on Android"
published: true
description: "A hands-on guide to INT4 key cache quantization, sliding window eviction, and memory-mapped spilling that fits Llama 3.2 3B into 2 GB RAM on Android with minimal quality loss."
tags: android, kotlin, mobile, architecture
canonical_url: https://blog.mvpfactory.co/kv-cache-quantization-llama-3b-android
---

## What We Will Build

In this workshop, I will walk you through the memory architecture that lets you run Llama 3.2 3B conversational inference inside a 2 GB RAM budget on Android. We are not touching model weights here — we are attacking the **KV cache**, the silent memory killer that most teams overlook entirely.

By the end, you will understand how to apply per-layer INT4/INT8 mixed quantization to key-value caches, implement a sliding window eviction policy with flash-backed spilling, and go from crashing after 3-4 conversation turns to sustaining 12+ turns on Snapdragon 8 Gen 3 and Tensor G4 hardware.

Let me show you a pattern I use in every on-device inference project.

## Prerequisites

- Familiarity with transformer attention and KV caches at a conceptual level
- An Android device with Snapdragon 8 Gen 3 or Tensor G4 (or emulator for code exploration)
- [llama.cpp](https://github.com/ggerganov/llama.cpp) built for Android (NDK toolchain)
- A Q4_K_M quantized Llama 3.2 3B model file

## Step 1: Understand Why the KV Cache Is Your Real Problem

A Q4_K_M quantized Llama 3.2 3B sits around 1.6–1.8 GB on disk. Load it, generate a few hundred tokens, and your process creeps past the 2 GB mark. The model did not grow. The KV cache is quietly eating hundreds of megabytes in FP16.

Llama 3.2 3B uses grouped-query attention (GQA) with 8 KV heads shared across 32 query heads — a 4:1 grouping ratio. That already gives you a 4x reduction over standard multi-head attention. But even so, a 2048-token context window at FP16 precision requires **~224 MB** of KV cache across all 28 layers.

Stack that on top of a 1.7 GB model plus runtime overhead, and you blow past a 2 GB budget. That 224 MB is the margin between fitting and crashing.

> **The docs do not mention this, but** without GQA (a hypothetical 32-head MHA design), the FP16 KV cache would consume ~896 MB. GQA plus mixed quantization together represent a 90%+ reduction from that MHA baseline. But the honest comparison is against the GQA-aware FP16 figure of ~224 MB, since that is what Llama 3.2 3B actually uses.

## Step 2: Apply Mixed-Precision KV Cache Quantization

Here is the gotcha that will save you hours: key and value caches do not need the same precision. Key caches tolerate aggressive quantization far better than value caches. This asymmetry is the single most impactful optimization available after GQA itself.

Use INT4 for keys and INT8 for values. Here is the math with GQA-aware 8 KV heads:

| Cache Component | Precision | Per-Token Per-Layer | 2048 Context (28 Layers) |
|---|---|---:|---:|
| Keys (baseline) | FP16 | 2,048 B | 112 MB |
| Values (baseline) | FP16 | 2,048 B | 112 MB |
| Keys (quantized) | INT4 | 512 B | 28 MB |
| Values (quantized) | INT8 | 1,024 B | 56 MB |
| **Total baseline** | FP16 | 4,096 B | **224 MB** |
| **Total optimized** | Mixed | 1,536 B | **84 MB** |

That is a **62% reduction** — from 224 MB down to 84 MB — without touching a single model weight.

With llama.cpp, this is a flag away:

bash
--cache-type-k q4_0 --cache-type-v q8_0


## Step 3: Implement Sliding Window Eviction + Flash Spilling

For multi-turn conversations that exceed the context window, you need an eviction policy. Here is the minimal setup to get this working: keep a fixed sliding window of the most recent 1536 tokens, combined with a "sink" of the first 64 tokens to preserve system prompt attention anchors. This keeps the active cache bounded.

Memory-mapped cache spilling to flash storage handles earlier turns. On Android, memory-map a file in the app's internal storage and write evicted KV pairs as quantized blocks:

kotlin
// Simplified cache spilling on Android
val cacheFile = File(context.cacheDir, "kv_spill.bin")
val channel = RandomAccessFile(cacheFile, "rw").channel
val mappedBuffer = channel.map(
FileChannel.MapMode.READ_WRITE, 0, MAX_SPILL_SIZE
)
// Evicted INT4 key blocks written directly to mapped region
mappedBuffer.put(quantizedKeyBlock)


When the model's attention pattern needs older context, the OS pages it back transparently. Flash reads on UFS 4.0 storage (standard on Snapdragon 8 Gen 3 devices) clock sequential reads at 4.2 GB/s — more than fast enough for occasional cache page-ins without perceptible latency.

## Step 4: Validate on Real Hardware

All benchmarks run on llama.cpp (commit `b4011`) with Q4_K_M model weights. Decode benchmarks use a 512-token prompt with 256-token generation, averaged over 10 runs. Ambient temperature held at 24°C; devices on a ventilated surface with screens off.

| Metric | SD 8 Gen 3 (FP16 KV) | SD 8 Gen 3 (Mixed KV) | Tensor G4 (FP16 KV) | Tensor G4 (Mixed KV) |
|---|---:|---:|---:|---:|
| Peak RSS (MB) | 2,100 | 1,920 | 2,130 | 1,950 |
| Tokens/sec (decode) | 8.2 | 9.4 | 6.8 | 7.9 |
| MMLU (5-shot) | 62.4 | 62.1 | 62.4 | 62.0 |
| MT-Bench (avg) | 7.62 | 7.58 | 7.62 | 7.55 |
| Max conversation turns (2 GB cap) | 4 | 12+ | 3 | 10+ |

Quality degradation on MMLU is under 0.5 points. MT-Bench scores stay within noise. The operational win is what matters: you go from crashing after a handful of turns to sustaining **12+ turn conversations** within budget. Token throughput also improves — smaller caches mean fewer cache misses and better memory bandwidth utilization.

## Step 5: Handle Thermal Throttling

Running sustained inference on-device generates real thermal load. On Snapdragon 8 Gen 3, sustained workloads trigger thermal throttling within 90 seconds. Query the Android Thermal HAL to detect approaching thresholds:

kotlin
val thermalHeadroom = powerManager.getThermalHeadroom(FORECAST_SECONDS)
if (thermalHeadroom < THROTTLE_THRESHOLD) {
// Insert brief pause between generation bursts
delay(COOLDOWN_MS)
}


On Tensor G4, Google's adaptive thermal framework is more aggressive. I have found that voluntarily targeting 70% of peak throughput avoids the cliff-edge drops that thermal governors impose. Long profiling sessions at the desk are where tools like [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) earn their keep — those sustained thermal benchmarking runs are exactly when you forget to move for two hours straight.

Memory-mapped spilling requires careful lifecycle management. Tie your mapped buffers to a foreground service or ViewModel scope to avoid leaks when the system reclaims your process.

## Gotchas and Common Mistakes

1. **Optimizing against phantom baselines.** Llama 3.2 3B's 8 KV heads already compress the cache 4x versus full MHA. Your true FP16 baseline is ~224 MB, not ~900 MB. Build your memory budget from the correct starting point.

2. **Quantizing keys and values identically.** Value caches are more sensitive to precision loss than key caches. INT4 for both will degrade quality noticeably. Use INT4 keys with INT8 values.

3. **Ignoring cache lifecycle on Android.** If your mapped buffers outlive the component that created them, you leak memory and file handles. Scope them properly.

4. **Skipping the thermal story.** Your benchmark numbers are meaningless if thermal throttling kicks in during real usage. Always profile sustained, not burst, performance.

## Wrapping Up

The core takeaway: **quantize KV caches asymmetrically** (INT4 keys, INT8 values), **bound your active cache** with sliding window eviction, and **spill to flash** for multi-turn persistence. This single architectural pattern recovers 62% of KV cache memory with sub-0.5-point quality impact, turning a crashing demo into a shipping product.

Do the real math with GQA. Start from the correct ~224 MB baseline. And build your memory budget before you build your features.

PostgreSQL Generated Columns and Expression Indexes for Multi-Tenant SaaS

SoftwareDevs mvpfactory.io — Mon, 15 Jun 2026 14:11:22 +0000

---
title: "Workshop: Cut P99 Latency 80% with PostgreSQL Generated Columns"
published: true
description: "A hands-on guide to eliminating runtime JSONB parsing in multi-tenant SaaS using PostgreSQL STORED generated columns, GIN expression indexes, and partial indexes scoped per tenant."
tags: postgresql, architecture, api, performance
canonical_url: https://blog.mvpfactory.co/postgresql-generated-columns-cut-p99-latency-80
---

## What We Will Build

In this workshop, we will take a slow multi-tenant SaaS query that parses JSONB on every read and transform it into an indexed, pre-computed lookup that cuts P99 latency by 80%. You will walk away with a migration strategy that works on PG 16+ without table rewrites or downtime.

Let me show you a pattern I use in every project that stores tenant config as JSONB.

## Prerequisites

- PostgreSQL 12+ (PG 16+ recommended for zero-downtime migrations)
- A table with a JSONB column used in `WHERE` clauses
- Basic familiarity with `EXPLAIN ANALYZE` and index types

---

## Step 1: Spot the Problem

Here is the gotcha that will save you hours. If your hot path looks like this, you have a scaling problem waiting to happen:

sql
-- This runs on every request. Every tenant. Every time.
SELECT * FROM orders
WHERE tenant_id = 'acme'
AND (config->>'shipping_tier')::int >= 3
AND config @> '{"region": "us-east"}';


At 10 tenants, nobody notices. At 1,000 tenants with millions of rows, your P99 latency graph looks like a hockey stick. The database is re-parsing and casting JSON strings on every single read.

| Metric | Raw JSONB query | With generated columns + GIN |
|--------|----------------|------------------------------|
| P50 latency | 12 ms | 4 ms |
| P99 latency | 210 ms | 38 ms |
| CPU per query | High (parse + cast) | Minimal (index scan) |
| Write overhead | None | ~5-8% per INSERT/UPDATE |
| Storage overhead | None | ~10-15% per indexed column |

A small write-time cost eliminates massive read-time waste. That is the whole trade.

## Step 2: Add STORED Generated Columns

PostgreSQL 12+ supports `GENERATED ALWAYS AS ... STORED` columns. These compute a value from other columns at write time and persist it on disk.

sql
ALTER TABLE orders
ADD COLUMN shipping_tier int
GENERATED ALWAYS AS ((config->>'shipping_tier')::int) STORED,
ADD COLUMN region text
GENERATED ALWAYS AS (config->>'region') STORED;


Now `shipping_tier` and `region` are real, typed, indexable columns. Computed once on write, never parsed again on read.

## Step 3: Add a GIN Expression Index on JSONB

For queries that need to match complex JSONB patterns across varying tenant schemas, GIN expression indexes still work well:

sql
CREATE INDEX idx_orders_config_gin ON orders
USING GIN (config jsonb_path_ops);


The docs do not mention this, but a single global GIN index across all tenants becomes bloated and slow. You need to combine it with partial indexes.

## Step 4: Create Partial Indexes Scoped Per Tenant

Here is the minimal setup to get this working for your highest-volume tenants. Scoped partial indexes shrink the index size and improve scan performance dramatically:

sql
CREATE INDEX idx_orders_acme_tier ON orders (shipping_tier)
WHERE tenant_id = 'acme' AND shipping_tier >= 3;


This index is tiny, fits in memory, and serves only the queries that matter. A 50 KB partial index that fits in L2 cache will always outperform a 2 GB global index.

## Step 5: Detect and Drop Dead Indexes

Partial indexes accumulate. Teams create them, forget them, and wonder why `VACUUM` takes forever. `pg_stat_user_indexes` exposes exactly what you need:

sql
SELECT schemaname, relname, indexrelname,
idx_scan, idx_tup_read, idx_tup_fetch,
pg_size_pretty(pg_relation_size(indexrelid)) AS size
FROM pg_stat_user_indexes
WHERE idx_scan = 0
AND schemaname = 'public'
ORDER BY pg_relation_size(indexrelid) DESC;


Any index with `idx_scan = 0` after a reasonable observation window is dead weight. Drop it. On one system I worked on, removing 40+ unused partial indexes reclaimed 12 GB and cut `autovacuum` duration by 35%.

## Step 6: The PG 16+ Migration (Zero Downtime)

Before PG 16, adding a STORED generated column triggered a full table rewrite. For tables with hundreds of millions of rows, that is a non-starter. PG 16 changed this for immutable (non-volatile) generated expressions.

sql
-- Step 1: Add generated column (PG 16+ -- no rewrite for immutable expressions)
ALTER TABLE orders
ADD COLUMN region text
GENERATED ALWAYS AS (config->>'region') STORED;

-- Step 2: Create index concurrently (no locks)
CREATE INDEX CONCURRENTLY idx_orders_region ON orders (region);

-- Step 3: Add tenant-scoped partial indexes for top tenants
CREATE INDEX CONCURRENTLY idx_orders_acme_region
ON orders (region) WHERE tenant_id = 'acme';

-- Step 4: Update queries to use the generated column
-- Old: WHERE config->>'region' = 'us-east'
-- New: WHERE region = 'us-east'


| PG version | ADD COLUMN (generated, STORED) | Downtime risk |
|------------|-------------------------------|---------------|
| 12-15 | Full table rewrite | High for large tables |
| 16+ | No rewrite (immutable expr) | Minimal |

That is the difference between a 3-hour maintenance window and a zero-downtime deploy.

---

## Gotchas

- **Write amplification**: Generated columns add storage and write-time cost. Skip them when the JSONB field changes on nearly every request.
- **Batch-only fields**: If you only need the extracted value in batch or offline workloads, a generated column is overkill — just parse at batch time.
- **Volatile expressions**: Generated column expressions must be immutable. If your expression depends on external state or is volatile, PostgreSQL will reject it.
- **Index sprawl**: Partial indexes per tenant are powerful, but they accumulate silently. Check `pg_stat_user_indexes` quarterly and cull anything with zero scans.
- **PG version matters**: On PG 12-15, adding a STORED generated column rewrites the entire table. Upgrade to PG 16+ before migrating large tables, or plan for a maintenance window.

---

## Conclusion

Audit your hot queries for runtime JSONB parsing. If you see `->>` casts or `@>` operators in your `pg_stat_statements` top-10 by total time, those are candidates for generated columns. Scope partial indexes per tenant for your highest-volume accounts, clean up dead indexes with `pg_stat_user_indexes`, and upgrade to PG 16+ to make the migration a routine `ALTER TABLE` you can ship without a maintenance window.

The pattern is straightforward: push computation to write time, index the result, scope per tenant. Your P99 will thank you.

Apple Foundation Models SDK with Claude Code: Building Hybrid On-Device/Cloud AI Pipelines for iOS Apps in Swift

SoftwareDevs mvpfactory.io — Mon, 15 Jun 2026 07:45:08 +0000

---
title: "Hybrid AI Pipelines: Apple On-Device + Claude Cloud"
published: true
description: "Build a tiered AI inference pipeline in Swift — route simple tasks to Apple's on-device models, escalate complex reasoning to Claude API, with an adapter pattern that keeps your feature layer clean."
tags: swift, ios, architecture, mobile
canonical_url: https://blog.mvpfactory.co/hybrid-ai-pipelines-apple-on-device-claude-cloud
---

## What We're Building

Let me show you a pattern I use in every project that ships AI features on iOS. We're going to architect a **tiered inference pipeline** that routes simple tasks — classification, short extraction — to Apple's Foundation Models framework on-device, and escalates complex reasoning to Claude's API. The glue is a protocol-based adapter so your feature layer never knows which provider answered.

By the end, you'll have: a unified `AIProvider` protocol, an intelligent router, token budget management, and Combine-based streaming that keeps your UI responsive regardless of provider.

## Prerequisites

- Xcode with Swift 6+
- A device with Apple Silicon (on-device inference requirement)
- An Anthropic API key for Claude access
- Familiarity with Swift concurrency (`async/await`, `AsyncThrowingStream`)

## Step 1: Define the Provider Protocol

The core abstraction is a protocol that both providers conform to. This is what makes swapping providers a config change instead of a rewrite.

swift
protocol AIProvider {
func generate(prompt: String, maxTokens: Int) async throws -> String
func stream(prompt: String) -> AsyncThrowingStream
var estimatedCapabilityTier: CapabilityTier { get }
}

enum CapabilityTier: Int, Comparable {
case basic = 0 // Classification, short extraction
case standard = 1 // Summarization, simple generation
case advanced = 2 // Multi-step reasoning, long-form analysis
}


Your on-device provider wraps Apple's `LanguageModelSession`. Your cloud provider wraps the Anthropic SDK. Same interface, different engines.

## Step 2: Build the Router

Here is the minimal setup to get this working. The router checks task complexity and token estimates, then picks a provider:

swift
struct InferenceRouter {
let onDevice: AIProvider
let cloud: AIProvider

func route(task: AITask) async throws -> String {
    if task.requiredTier <= onDevice.estimatedCapabilityTier
        && task.estimatedTokens < 512 {
        return try await onDevice.generate(
            prompt: task.prompt, maxTokens: task.estimatedTokens
        )
    }
    return try await cloud.generate(
        prompt: task.prompt, maxTokens: task.estimatedTokens
    )
}

}


The split is intuitive: anything that fits a short context and needs a quick answer (sentiment, entity extraction, autocomplete) — on-device wins on latency (~50–200ms), cost (zero), and privacy (data never leaves device). The moment you need chain-of-thought reasoning, long document analysis, or nuanced generation — send it to Claude.

## Step 3: Add Combine-Based Streaming

Both providers can stream tokens. Wrapping them in a Combine pipeline keeps your UI responsive regardless of which provider is active:

swift
func streamResponse(for task: AITask) -> AnyPublisher {
let stream = router.routeStreaming(task: task)

return stream
    .receive(on: DispatchQueue.main)
    .scan("") { accumulated, chunk in accumulated + chunk }
    .eraseToAnyPublisher()

}


Your SwiftUI view subscribes to a single publisher. It doesn't care whether tokens come from Apple Silicon or Claude's API.

## Step 4: Guard Your Cloud Budget

In production, you need guardrails. This actor prevents runaway costs:

swift
actor TokenBudgetManager {
private var dailyCloudTokensUsed: Int = 0
private let dailyLimit: Int = 100_000

func canUseCloud(estimatedTokens: Int) -> Bool {
    dailyCloudTokensUsed + estimatedTokens <= dailyLimit
}

func recordUsage(_ tokens: Int) {
    dailyCloudTokensUsed += tokens
}

}


When the budget runs out, the router gracefully degrades to on-device only. Users still get responses — just simpler ones. Far better than a hard failure or a surprise bill.

## Step 5: Enforce Privacy Boundaries

The docs don't mention this, but this is the architectural decision that matters most. Define a clear data classification in your routing logic, **not** in your feature code:

- **Tier 1 (on-device only):** health data, financial records, personal messages — anything covered by privacy regulations.
- **Tier 2 (cloud-eligible):** generic content generation, public data analysis, non-personal queries.

The feature layer asks for "summarize this text." The router checks data classification *before* picking a provider. Privacy enforcement stays centralized and auditable.

## Gotchas

- **Don't route by gut feeling.** Classify each AI task by complexity requirements. Let the router decide based on token estimates, reasoning depth, and privacy constraints — not hardcoded provider choices scattered through your codebase.
- **On-device context windows are constrained by device memory.** Claude supports up to 200K tokens. If you're sending long documents to on-device and getting truncated results, that's why.
- **Apple's `@Generable` macro handles structured output on-device**, while Claude uses tool use and JSON mode. Your adapter needs to normalize both into the same response shape.
- **Design for offline-first.** Treat on-device as your baseline and cloud as your escalation path. When the network is unavailable or the token budget is spent, your app still works.

## Wrapping Up

Build the adapter layer now — even if you only use one provider today. The protocol-based abstraction is nearly free and saves you a rewrite later. The hybrid approach is the right default for most iOS apps shipping AI features because you're optimizing across latency, cost, privacy, and capability simultaneously instead of picking one axis and hoping the others work out.

Start with the `AIProvider` protocol, get on-device working first, then add Claude as your escalation path. You'll have an AI layer that's resilient, cost-aware, and privacy-respecting from day one.

Structured Output Grammars for On-Device LLMs

SoftwareDevs mvpfactory.io — Fri, 12 Jun 2026 13:47:46 +0000

---
title: "Structured Output Grammars for On-Device LLMs on Android"
published: true
description: "A hands-on guide to using GBNF grammars in llama.cpp to guarantee valid JSON from on-device LLMs on Android — no retry loops, no post-processing."
tags: kotlin, android, architecture, mobile
canonical_url: https://blog.mvpfactory.co/structured-output-grammars-on-device-llms-android
---

## What We Will Build

By the end of this tutorial, you will have a working setup where your on-device LLM on Android produces **structurally valid JSON every single time** — not 7 out of 10, not 9 out of 10, but 10 out of 10. We will write a custom GBNF grammar, wire it into llama.cpp through JNI, and watch grammar-guided sampling eliminate the retry loops you have been writing around malformed model output.

Let me show you a pattern I use in every project that runs inference on-device.

## Prerequisites

- Android project with llama.cpp integrated via its JNI bridge
- A quantized GGUF model (Q4_K_M or similar) deployed to device
- Familiarity with Kotlin and basic BNF notation
- A Snapdragon 8 Gen 3 (or comparable) device for realistic benchmarks

## Step 1: Understand Why This Is a Sampling Problem

Here is the gotcha that will save you hours: most teams treat malformed LLM output as a post-processing problem. They wrap inference in a try-parse-retry loop. On mobile, where every millijoule counts, that is the wrong approach entirely.

The fix belongs in the **decoder**. GBNF (GGML BNF) is a grammar format supported natively by llama.cpp. At each token generation step, the sampler checks which tokens are valid continuations given the current grammar state. Invalid tokens get their logits masked to negative infinity before softmax. The model literally cannot produce an invalid sequence.

The pipeline:

Logits → Grammar Mask → Temperature → Top-K/Top-P → Token Selection


This is not regex validation after the fact. The grammar creates a finite-state automaton that walks forward with each generated token, pruning the search space in real time.

## Step 2: Write a Custom GBNF Grammar

Suppose your API expects this schema from the model:

json
{"intent": "string", "confidence": 0.0, "entities": [{"name": "string", "type": "string"}]}


Here is the minimal setup to get this working — a GBNF grammar that locks every field name as a literal:

bnf
root ::= "{" ws "\"intent\":" ws string "," ws "\"confidence\":" ws number "," ws "\"entities\":" ws entities ws "}"
string ::= "\"" [a-zA-Z0-9_ ]+ "\""
number ::= "0" "." [0-9]+
entities ::= "[" ws (entity ("," ws entity))? ws "]"
entity ::= "{" ws "\"name\":" ws string "," ws "\"type\":" ws string ws "}"
ws ::= [ \t\n]


The model has zero freedom to hallucinate keys like `"conf"` or `"entity_list"`. It fills in values; the grammar enforces structure. Write grammars that match your **exact schema**, not generic JSON. A generic `json.gbnf` guarantees valid JSON but not valid *responses*. Schema-specific grammars give you both.

## Step 3: Integrate with Kotlin via JNI

The llama.cpp Android example exposes grammar support through its JNI bridge. Pass the grammar string when you configure the sampler:

kotlin
external fun setupGrammarSampler(
contextPtr: Long,
grammarString: String
): Long

// In your inference wrapper
val grammar = assets.open("schema.gbnf").bufferedReader().readText()
val samplerPtr = setupGrammarSampler(ctxPtr, grammar)


On the C++ side, the grammar is parsed once into a `llama_grammar` instance and reused across tokens within a single generation. The per-token cost is the automaton state advance and logit masking — both O(V) where V is vocabulary size. On 32K-vocab models, that adds roughly 0.2 ms per step.

## Step 4: Measure the Real-World Tradeoff

Here are representative numbers from a Q4_K_M quantized 7B model running on a Snapdragon 8 Gen 3:

| Metric | Unconstrained | Grammar-guided | Delta |
|---|---|---|---|
| Tokens/sec (decode) | 12.4 t/s | 10.8 t/s | -12.9% |
| Time-to-first-token | 280 ms | 295 ms | +5.4% |
| Valid JSON rate | ~72% | 100% | +28pp |
| Avg retries needed | 0.4 | 0 | -100% |
| Effective latency (incl. retries) | 1,480 ms | 1,120 ms | -24.3% |

You pay roughly 13% on raw decode speed, but you eliminate retries entirely. Net effective latency drops by nearly a quarter. On battery-constrained devices, avoiding redundant inference passes matters even more than the raw throughput number suggests.

## Gotchas

**Token boundary misalignment on quantized models.** The docs do not mention this, but it will bite you in production. Consider generating `"confidence": 0.85`. A BPE tokenizer might encode `0.85` as `["0", ".", "8", "5"]` or as `["0", ".85"]` depending on the merge table. Aggressive quantization (Q2_K, Q3_K_S) shifts probability mass in ways that interact poorly with grammar masking.

What this looks like in practice:

- Numeric values truncated at unusual boundaries (`0.` followed by EOS)
- Strings ending mid-token because no valid continuation exists in the grammar
- Repeated whitespace tokens when the grammar allows `ws` as a fallback

**The fix: defensive grammar design.** For numeric fields, allow broader patterns than your schema strictly requires, then validate semantically in Kotlin after parsing:

bnf
number ::= "-"? [0-9]+ ("." [0-9]+)? ([eE] [+-]? [0-9]+)?


Let the grammar guarantee structure. Your application layer handles meaning. Keep those responsibilities separate and you will save yourself a lot of debugging.

**Do not use generic JSON grammars.** A generic grammar guarantees valid JSON but not valid responses. If the model hallucinates a key name that is syntactically valid JSON, a generic grammar will happily allow it. Lock your field names as literals.

## Wrapping Up

Move validation into the decoder. The 10-15% decode overhead pays for itself by removing retry loops, and effective latency drops 20%+ on real workloads. Write schema-specific GBNF grammars with literal field names. Design those grammars defensively for quantized models — permissive value patterns, strict structural rules. The grammar handles structure; Kotlin handles semantics.

That separation of concerns is what makes this pattern production-ready on mobile.

Replacing Your Kubernetes Cluster with a Single SQLite-Backed Binary

SoftwareDevs mvpfactory.io — Fri, 12 Jun 2026 08:30:17 +0000

---
title: "Replace Kubernetes with SQLite: The $5 VPS Architecture for Early-Stage SaaS"
published: true
description: "A hands-on guide to embedding SQLite with Litestream continuous replication as your production backend — WAL-mode tuning, S3 recovery, single-binary deployment, and when to migrate to PostgreSQL."
tags: architecture, devops, cloud, postgresql
canonical_url: https://blog.mvpfactory.co/replace-kubernetes-with-sqlite-the-5-dollar-vps-architecture
---

## What We Will Build

Let me show you a pattern I use in every early-stage project: a single compiled binary embedding SQLite in WAL mode, continuously replicated to S3 via Litestream, deployed to a $5 VPS with zero container orchestration. By the end of this tutorial, you will have a production-ready architecture that handles thousands of concurrent reads, costs under $11/month, and recovers from disaster in seconds.

No Kubernetes. No Helm charts. No 2 AM pages about pod scheduling.

## Prerequisites

- A VPS ($5-$10/month from any provider — Hetzner, DigitalOcean, etc.)
- A Go or Rust toolchain for building your binary
- An S3-compatible bucket (AWS S3, Backblaze B2, MinIO)
- [Litestream](https://litestream.io/) installed on your VPS
- Basic familiarity with SQLite and systemd

## Step 1: Understand What You Are Replacing

Here is the gotcha that will save you months of over-engineering. Most early-stage teams spin up Kubernetes on day one for 50 daily active users. Look at the real cost comparison:

| Component | Kubernetes Stack | SQLite + Litestream |
|---|---|---|
| Compute | 3-node cluster (~$150/mo) | Single $5-$10 VPS |
| Database | Managed PostgreSQL (~$15-$50/mo) | Embedded SQLite ($0) |
| Replication/Backups | Automated DB backups (~$5/mo) | Litestream to S3 (~$0.50/mo) |
| Load Balancer | Cloud LB (~$18/mo) | Caddy reverse proxy ($0) |
| Container Registry | Registry + CI/CD pipeline | `scp` a binary |
| **Monthly Total** | **$200-$450** | **$6-$11** |

That is a 20-40x cost reduction with far less stuff that can break.

## Step 2: Set Up WAL-Mode Tuning for Concurrent Reads

SQLite's default journal mode serializes everything. WAL (Write-Ahead Logging) mode unlocks concurrent reads while writes remain serialized — exactly what a typical SaaS read-heavy workload needs.

Here is the minimal setup to get this working. Set these pragmas at connection time:

sql
PRAGMA journal_mode=WAL;
PRAGMA busy_timeout=5000;
PRAGMA synchronous=NORMAL;
PRAGMA cache_size=-20000; -- 20MB cache
PRAGMA foreign_keys=ON;
PRAGMA wal_autocheckpoint=1000;


The docs do not mention this, but `synchronous=NORMAL` instead of `FULL` is the key performance lever. You lose durability guarantees only in catastrophic OS-level crashes — and Litestream's continuous S3 replication already covers that failure mode. In benchmarks, this combination handles 10,000+ reads/second and 1,000+ writes/second on modest hardware.

## Step 3: Configure Litestream for Continuous Replication

Litestream does not take periodic snapshots. It streams WAL frames continuously with sub-second replication lag, giving you point-in-time recovery granularity measured in seconds.

The architecture looks like this:

plaintext
[Your Binary (Go/Rust/etc.)]
│
▼
[SQLite - WAL Mode]
│
▼
[Litestream] ──stream──▶ [S3 Bucket]
│
▼
Point-in-time recovery from any snapshot


Recovery is a single command:

bash
litestream restore -o /data/app.db s3://your-bucket/app.db


Your entire database restores from S3 in seconds for typical early-stage datasets. Compare that to restoring a PostgreSQL dump or waiting for a managed database snapshot to provision.

## Step 4: Deploy with a Single Binary

Here is your entire production deployment script:

bash

!/bin/bash

scp myapp user@vps:/opt/myapp/myapp-new
ssh user@vps 'systemctl stop myapp && \
mv /opt/myapp/myapp-new /opt/myapp/myapp && \
systemctl start myapp'


No container registry. No image layers. No pod scheduling. No rollout strategies. Your CI pipeline builds a binary, copies it to the server, and restarts the service. Downtime is under one second.

For an early-stage SaaS with a handful of users, this is honestly better than the alternative. Every layer of orchestration you remove is a layer that cannot page you at 3 AM.

## Step 5: Define Your Migration Triggers Upfront

This is the section most advocates skip, and it is the most important one. SQLite is single-writer. Know your ceilings before you start:

| Metric | SQLite Comfortable Range | When to Migrate |
|---|---|---|
| Concurrent write transactions | < 50/sec sustained | When you consistently exceed this |
| Database file size | < 10 GB | When queries over large tables slow |
| Horizontal read scaling | Never | When a single box cannot serve reads |
| Multi-region requirements | Not feasible | When latency mandates geo-distribution |

The write concurrency ceiling is what you will hit first. A SaaS handling user-generated content (form submissions, API calls, event tracking) typically crosses the discomfort zone around 500-1,000 monthly active users generating write-heavy workloads. Read-heavy products like dashboards or content delivery can push well beyond that.

The migration path is well-trodden: export to PostgreSQL using `pgloader`, update your queries (SQLite's SQL dialect is 95% compatible), and deploy. Budget a weekend.

## Gotchas

1. **Forgetting WAL mode.** Without it, SQLite serializes all reads and writes. Always set `journal_mode=WAL` before anything else.
2. **Setting `synchronous=FULL` out of fear.** With Litestream replicating continuously, `NORMAL` is safe and dramatically faster. Save `FULL` for contexts without replication.
3. **No migration trigger defined.** Decide upfront: "When sustained writes exceed 50/sec, we start the PostgreSQL migration." Without a concrete number, you will either migrate too early (wasting time) or too late (degrading user experience).
4. **Over-engineering the deployment.** Resist the urge to containerize a single binary. `scp` and `systemctl restart` is your friend until traffic proves otherwise.
5. **Ignoring cognitive load.** Simple infrastructure matters more than you think. When your entire backend is one process and one file, you spend less mental energy on ops and more on the product. During long architecture sessions, I keep [HealthyDesk](https://play.google.com/store/apps/details?id=com.healthydesk) running to remind me to step away from the desk — turns out your deployment architecture matters less than your spine's architecture.

## Conclusion

Start with SQLite + Litestream if you are pre-product-market-fit. The $200+/month you save compounds, and the operational simplicity lets a solo developer or small team move faster. Set WAL mode, tune your pragmas, point Litestream at an S3 bucket, and treat your binary as the artifact — not a container image.

Add complexity only when traffic demands it, and let the data drive that decision instead of your anxiety about scale. Scale your ambition first. Scale your servers when the metrics tell you to.