DEV Community: alfchee

allow_sales: false

alfchee — Wed, 09 Sep 2026 02:33:32 +0000

allow_sales: false

That's it. That's the reason I can't charge money for my software.

Not a rejected application. Not a compliance review that went badly. A boolean field, for country code NI, in a payment provider's configuration. Signups blocked, transactions blocked, at the API layer. Nothing to appeal, because nothing was ever considered.

I want to write down what I found, because I went looking for this article before I wrote it and it doesn't exist.

The app, briefly, and the elephant

I built a shared personal-finance tracker for couples. My wife and I needed one, Honeydue isn't available in Nicaragua, and every alternative assumes you can connect your bank through Plaid — which covers the US and parts of Europe and nothing where I live.

Yes: another personal finance app. I know. It's the modern to-do list, the thing every developer builds now that an agent can scaffold a CRUD with charts in an afternoon. If you came here for that, there's nothing for you.

The app isn't the story. The story starts when I tried to charge for it.

The sentence every tutorial ends with

"Now just add Stripe and you're done."

I've read that sentence in maybe two hundred tutorials, and I never once noticed that it's a geographic claim disguised as a technical step.

Stripe isn't available to sellers in Nicaragua. Fine — that's what merchant-of-record platforms are for. An MoR sells to your customer on your behalf, absorbs the cross-border sales tax problem, and pays you out. That's the standard answer for developers outside the US and EU, and there's a healthy market of them now.

So I worked through the market. Here's what I found.

Five providers

Dodo Payments

I created the account first and read the docs second, which was my mistake.

Nicaragua isn't on their accepted-countries list. What made this land was the shape of the omission: Belize, Costa Rica, El Salvador, Guatemala, Honduras and Panama are all there. Every country I share a border or a region with. Not mine.

Their eligibility rule is the important detail, and it's the one that kills the obvious workaround:

Eligibility is based on the country that issued the government-issued identity document you verify with, not on where your company is registered or where you pay tax.

So "just incorporate in Panama" doesn't work. For a registered entity, every director and beneficial owner has to verify with an ID from a listed country. A Panamanian company with me as its sole owner fails the same check.

Their failure paths are also one-shot, which matters more than it sounds:

Identity verification does not reopen automatically after a failure — you have to email support and ask
A form marked Failed gets you one appeal, and the appeal decision is final
And: "If your business could not be supported for reasons of location or product category, an appeal cannot change that outcome" Read those together. If I had submitted the KYC form and let it fail on country, I'd have burned my single appeal on a question the appeal process explicitly cannot answer. Their docs actually tell you to ask first — buried in an info box that I'm glad I read:

Not sure how this applies to you? Contact support before you begin your integration, so you don't build against an outcome that cannot be approved.

Creem

Creem publishes a list of 32 countries it can't accept payments from — merchants or customers. Nicaragua is on it.

Here's the company it keeps: Afghanistan, Belarus, Cuba, Iran, Iraq, Libya, North Korea, Russia, Somalia, Sudan, Syria, Venezuela, Yemen.

That list told me more than the other four providers combined. This isn't a coverage gap that someone will get around to filling. It's a risk classification, and my country is in a bucket with active war zones and comprehensively sanctioned states.

Polar

Polar's payments are global — "except from countries with US sanctions (Cuba, Russia, Iran, North Korea, and Syria)." Nicaragua isn't in that list, which was briefly encouraging.

Then I read the payouts section. Polar pays out through Stripe Connect Express to a specific list of countries. Costa Rica, El Salvador, Guatemala, Panama. No Nicaragua, no Honduras.

So customers in Nicaragua can pay Polar. Polar cannot pay me. Being able to take money you can never collect is a strange kind of "supported".

Lemon Squeezy

Two payout methods: bank transfer and PayPal.

The bank-payout list excludes Nicaragua. The PayPal branch stays theoretically open, since they pay out to PayPal in 200+ countries — but PayPal in Nicaragua has historically been send-only, and "you can pay with it but not get paid into it" is the local default for a lot of financial products.

Their eligibility rule is refreshingly blunt:

If you don't see your country listed by PayPal or in the list below, sorry, you won't be able to use Lemon Squeezy.

Moot anyway: since the Stripe acquisition, new merchant access is waitlist-only.

Paddle

Paddle was the last real candidate, and the most promising one — it's the only MoR I found that supports Payoneer as a payout method natively, which mattered because Payoneer is one of the few financial rails that reaches me.

I asked their assistant directly. The answer:

The knowledge sources explicitly list Nicaragua (NI) as a country where allow_sales is false, meaning Paddle blocks both signups and transactions from Nicaragua. [...] Paddle's API actively prevents signups from unsupported/sanctioned countries.

That's the field name. That's the whole conversation.

Five for five

Provider	Result
Dodo Payments	Nicaragua absent from accepted countries
Creem	Explicitly unsupported — listed with Cuba, Iran, Syria
Polar	Payments yes, payouts no
Lemon Squeezy	Bank list excludes it; invite-only anyway
Paddle	`allow_sales = false` at the API

The part I want other developers to understand

This is de-risking, not sanctions.

Nicaragua is not comprehensively sanctioned the way Cuba, Iran, North Korea and Syria are. US measures target named individuals and entities, not a blanket prohibition on Nicaraguan people receiving payment for software. There is no law that says a payment company can't onboard me.

What's happening is that every one of these providers sits on top of the same handful of acquirers, banks and card networks, and those upstream partners apply risk scoring built from FATF, OFAC and EU AML frameworks. When a country scores badly, the cheapest decision for everyone downstream is a boolean. Nobody in the chain has to decide I'm a risk. The chain just never gets to me.

Two consequences worth internalising:

It's a commercial decision, which means it's a conversation. A legal prohibition is a wall. A risk policy is a threshold, and thresholds have exceptions and appeals and humans behind them. That's why the country lists differ between providers who are otherwise near-identical, and why asking directly is worth doing even when the list says no.

It's invisible from the other side. Every payments tutorial, every "ship your SaaS in a weekend" course, every Stripe integration guide, is written by someone for whom this problem doesn't exist. Not out of malice — they genuinely don't know the layer is there. Which is exactly why nobody writes this down.

The workaround I didn't take

Every forum thread on this converges on the same two suggestions.

Use a US receiving account. Payoneer and Wise give you real US/EU account details in your own name. Enter those as your bank account and the country check appears to vanish.

Have someone abroad be the merchant. A friend or relative in a supported country signs up, receives the money, forwards it on.

Both are misrepresentation to the provider, and I want to be specific about why I said no, because "it's against the rules" isn't a real argument to someone who's been told no five times.

The failure mode is the worst shape available. It doesn't fail at signup, when failing costs you nothing. It fails at the first payout, or the first compliance review, or the first chargeback — with customer money already collected, an account you can no longer withdraw from, and subscribers whose payments you now have to unwind. Not being able to get paid is a problem. Being paid and then frozen while you owe refunds to real people is a worse one.

And for the second option, you've added someone else's identity, tax position and banking relationship to your risk surface, permanently.

The distinction I settled on: giving a provider a Payoneer account when they ask for one and you've told them where you live is completely fine — that's exactly why Paddle was my last hope. Using one to appear domiciled somewhere I'm not is a different thing entirely.

What I built instead

Here's the part that's actually useful if you're anywhere in this situation: you can build your entire billing domain against zero payment providers, and you should, because the answer to "which provider" may take months and shouldn't dictate your architecture.

One port

export interface PaymentProvider {
  readonly id: string;

  createCheckout(input: {
    householdId: string;
    planCode: string;
    idempotencyKey: string;
  }): Promise<{ redirectUrl?: string; clientToken?: string; reference: string }>;

  cancelSubscription(input: { subscriptionRef: string }): Promise<void>;

  // Authoritative state, for reconciliation against local records.
  getSubscription(input: { subscriptionRef: string }): Promise<ProviderSubscription>;

  // Verify and normalise an inbound webhook into internal domain events.
  parseWebhook(req: Request): Promise<BillingEvent[]>;
}

The critical design choice is BillingEvent. It's my vocabulary, not a vendor's:

type BillingEvent =
  | { type: "subscription.activated"; ref: string; planCode: string; periodEnd: Date }
  | { type: "payment.succeeded";      ref: string; amount: Money }
  | { type: "payment.failed";         ref: string; attempt: number }
  | { type: "subscription.cancelled"; ref: string; effectiveAt: Date }
  | { type: "subscription.expired";   ref: string };

Every adapter translates into that set. Entitlements, grace periods and dunning only ever see these five events. No stripe. or paddle. prefix reaches the domain layer, which means switching providers is writing one file.

One stub

export class StubPaymentProvider implements PaymentProvider {
  readonly id = "stub";
  constructor(private clock: Clock, private store: StubStore) {}

  // Test-only: drive the lifecycle deterministically.
  async advance(ref: string, to: SubscriptionStatus): Promise<BillingEvent[]> { /* ... */ }
  async simulateFailedRenewal(ref: string): Promise<BillingEvent[]> { /* ... */ }
  async redeliver(eventId: string): Promise<BillingEvent[]> { /* ... */ }
}

With an injectable clock, a 30-day trial and a 7-day grace period run in milliseconds. I can test the full path — trialing → active → past_due → grace → cancelled → expired, plus reactivation, duplicate webhooks and out-of-order delivery — with no provider, no sandbox, and no account.

Those are the parts of billing that are hardest to get right and most expensive to get wrong, and they're now covered by tests that ran before I knew who would process a payment.

The bug this design already caught

While building the entitlement helpers I wrote a function returning a feature limit from a config table. If the row was missing it returned NULL. In the trigger that enforced the limit, the comparison count >= limit against NULL evaluates to NULL — not true — so the insert proceeded.

A missing configuration row meant unlimited, silently, on the paid feature. Fail-closed isn't a slogan; coalesce it to zero.

I found that with the stub, in a test, months before any money existed. That's the argument for this approach better than anything I can say about hexagonal architecture.

Where I ended up

Local acquiring. My wife runs a business with a year of clean payment-processing history at a regional bank, so the route is a proper merchant account with the correct business classification — asked for openly, not borrowed.

It's a genuinely different deal, and worth knowing the shape of before you go looking for it. A local acquirer is not a merchant of record. I become the seller: sales tax is mine, invoicing is mine, chargebacks are mine. Which means cross-border selling stops being free — without an MoR absorbing it, selling a subscription into six countries makes me a tax filer in six countries.

So the paid tier launches domestic only. Not as an apology. It's the correct scope, and it happens to match who actually uses the app.

If you're reading this from a country like mine

Find the accepted-countries page before you build the pricing page. It's usually three clicks deep and it's the single highest-information document about your project's viability.
Ask compliance directly rather than inferring from a list. These are risk decisions, not laws. The answers differ between providers and sometimes between applicants.
Don't submit a KYC form you expect to fail. Appeals are often one-shot, and a location rejection can't be appealed anyway.
Design the provider boundary first. Then the answer, whenever it arrives, is one adapter instead of a rewrite.

5. Check whether your users can pay before checking whether you can be paid. Card penetration and recurring-charge support in your market may constrain the model more than any API does.

The code was never the hard part. An agent could have built the app.

The hard part is that "just add Stripe and you're done" is a sentence written by people for whom it's true, and if you're not one of them, nobody writes down what happens next.

So — here it is.

I'm a solution architect and DevOps engineer in Nicaragua. The app is a shared finance tracker for couples that my wife and I actually use; the repo is public because the interesting part is the decisions, not the CRUD. If you've hit this same wall from somewhere else, I'd like to hear how it went — the more of these that get written down, the less time the next person loses.

I Tested Three Vision Models on Catalog Images: OCR Was the Easy Part

alfchee — Tue, 01 Sep 2026 00:10:45 +0000

Converting a visual product catalog into structured, queryable data sounds simple on paper:

Send the page image to an OCR or Vision-Language Model (VLM).
Extract product codes and prices into JSON.
Ingest the rows into a production database.

Our hands-on benchmark with Mistral OCR, DeepSeek V4 Flash Vision Exp, and Qwen3-VL-32B-Instruct proved why that mental model breaks down in real life.

The hard problem was never character recognition. The real bottleneck was relationship preservation: linking the right 5-digit SKU to the correct variant, distinguishing campaign offers from list prices, and resolving multi-product shared-price blocks without human intervention.

Here is what we learned building an extraction pipeline for 138-page PDF catalogs, why raw OCR metrics are misleading, and how we architected a human-in-the-loop workflow that prevents business incidents.

1. Input Constraints: You Can't Upscale Missing Information

Our catalog consisted of 138 image-only pages, each embedded as a full-page JPEG capped at 150 DPI.

Our first architectural decision was straightforward: rendering the PDF at 300+ DPI was a waste of compute. Upscaling low-resolution assets doesn't create new signal; it just burns CPU cycles and expands token payloads.

Instead, our pipeline extracts the raw JPEG byte stream directly and feeds the original images to model endpoints.

The real challenge lay in the layout patterns:

Vertical product codes placed over low-contrast photography.
Campaign prices and list prices printed side by side.
Shared price blocks applying to multiple product codes.
Editorial pages with zero products that still trigger false positives.

If a pipeline transcribes every digit correctly but pairs a campaign price with the wrong SKU, it has generated corrupt catalog data.

2. The Benchmark: 3 Models on Complex Layouts

We evaluated three candidates across the same catalog fixtures:

Mistral OCR (mistral-ocr-latest): Document layout and text extraction.
DeepSeek V4 Flash Vision Exp: Vision extraction via OpenRouter ($0.00269/page actual).
Qwen3-VL-32B-Instruct: Structured vision extraction via OpenRouter ($0.00024/page actual).

All requests ran with deterministic settings (temperature=0). We captured raw payloads, latency, normalized JSON schemas, and per-page cost.

The Baseline Test: Page 94 (Shared Price Fixture)

We tested all three candidates against a tricky baseline: Page 94, featuring three SKUs (41367, 41368, 41369) sharing a single campaign price (C$339.00) and list price (C$570.00).

Model	SKU Exact Match	Campaign Price Exact Match	Shared-Price Resolution	Cost / Page
Mistral OCR	0% (0/3)	100% (1/1)	0% (0/1)	~$0.00350 (Est.)
Qwen3-VL-32B	0% (0/3)*	100% (1/1)	0% (0/1)	$0.00024 (Actual)
DeepSeek V4 Flash	0% (0/3)	100% (1/1)	0% (0/1)	$0.00269 (Actual)

*Note on Qwen3-VL below: the 0% score was caused by an evaluator schema mismatch, not an inference failure.

The takeaway: Every model caught the campaign price. But isolating rotated product codes and mapping them back to a shared parent price failed across automated runs. DeepSeek hallucinated nearby numbers (41347, 41348, 41349), while Mistral misclassified the list price as a second campaign price.

3. The Evaluator Is Part of the System Under Test

Our biggest engineering takeaway wasn't model accuracy—it was an evaluator contract bug.

During manual verification, Qwen3-VL had successfully identified product codes and prices. However, our automated test harness reported a 0% match rate because the scorer expected a JSON field named code, while Qwen returned product_code.

A single key mismatch turned a working extraction into a synthetic 0% metric on the dashboard.

Rule of thumb: In LLM/VLM evaluation pipelines, parser contracts are production code:

Standardize on a single canonical schema across all model adapters.
Preserve raw API payloads to debug evaluator bugs vs. model hallucinations.
Score entity relationships (Price ↔ SKU mapping), not just raw token overlap.

4. Operational Guardrails: Assistance Over Full Automation

Because a wrong price or mismatched SKU causes actual financial errors downstream, we rejected a 100% zero-touch ingestion pipeline. Instead, we established operating confidence bands:

≥ 98% Exact Match: Eligible for automated ingestion.
85% – 97% Match: Fast-track human review (side-by-side verification).
< 85% Match: Defer to manual entry if correction overhead exceeds drafting from scratch.

We selected Qwen3-VL-32B-Instruct for our pipeline due to its exceptional cost-to-performance ratio ($0.00024/page) and clean structured output, backing it with a human-in-the-loop review interface:

VLM extracts structured data into staging tables.
Reviewers inspect extracted fields side-by-side with the original catalog page.
Automated validation runs math sanity checks (e.g., verifying (List Price - Campaign Price) / List Price matches the printed badge discount within 1%).
Only human-approved rows are promoted to the active application database.

Key Takeaways for Engineers

OCR detects tokens; Vision extracts relationships. If your system cannot map a discount to its target SKU, high OCR precision is meaningless.
Respect the source ceiling. Don't waste latency upscaling low-DPI catalog scans—extract raw JPEG bytes directly.
Discount reconciliation is your best validation test. Cross-referencing percentage badges with arithmetic catches price swaps before human review.
Human-in-the-loop is a product feature, not technical debt. Building efficient side-by-side audit tools turns an imperfect 90% model into a 10x multiplier for operations.

How to Run a Real-Time API Across 10 Global Regions on GKE — Without Duplicating Your Manifests

alfchee — Fri, 17 Jul 2026 05:55:55 +0000

When you're building a real-time speech translation product, latency isn't a performance metric — it's the product itself. A 300ms round-trip to process an audio chunk is the difference between a conversation that feels natural and one that feels broken. That's the constraint that pushed us to design a multi-cluster, multi-region Kubernetes setup across 10 GCP regions.

This article walks through the full architecture: every layer, the decision behind each one, and the gotchas that will bite you if you're not careful. I'll use a fictional app called SpeakFlow — a real-time speech translation SaaS — to illustrate the patterns.

The Problem That Forces Multi-Region

SpeakFlow processes real-time audio in a tight loop: speech-to-text, translation, text-to-speech, under strict latency budgets. A user speaking in Tokyo, routed to a cluster in us-central1, adds ~150ms of pure network latency before any compute happens. For a real-time product, that's a broken experience.

The solution is obvious in theory: run the stack close to users. 10 GCP regions, covering every major market — North America, Europe (including a GDPR-dedicated Germany cluster), Asia Pacific, South America, Australia.

The engineering challenge: do it without creating 10 independently drifting copies of your infrastructure. One team, one source of truth, 10 regions.

The Stack

Before going into the how, here's what gets deployed to every region:

translate-api — main API, handles REST and WebSockets (Socket.IO)
translate-agg — aggregates real-time transcription events
translate-worker — async batch processing for offline audio jobs
manager-api — admin operations and organization management
dashboard-app — the React SPA served via nginx

Each runs in its own GKE cluster per region, backed by regional Memorystore Redis and a paired Azure Cognitive Services instance (STT, Translator, Language).

The dual-cloud pattern — GCP for compute, Azure for AI services — is a deliberate decision. Azure's speech and translation APIs are best-in-class for our use case. Running them in the same geographic zone as the GKE cluster keeps cross-cloud latency under 20ms.

The Core Pattern: One Base Manifest, Ten Overlays

The most important architectural decision is using Kustomize with a strict base/overlay separation.

prod.yml is the single source of truth for all deployments, services, HPAs, and security configurations across all 10 regions. It is never applied directly to a cluster. It defines everything that is identical everywhere: image tags, replica counts, resource limits, probe paths, security contexts, and all secrets that are shared across regions.

k8s/manifests/speakflow/multiregion/
├── prod.yml                ← base — READ ONLY, never apply directly
├── external-secret.yml     ← base ExternalSecret (ESO v1)
├── gateway-prod.yml        ← GKE Gateway + HTTPRoutes (once, not per-region)
└── per-region/
    ├── us-central1/kustomization.yml
    ├── europe-west2/kustomization.yml
    ├── europe-west3/kustomization.yml
    ├── northamerica-northeast1/kustomization.yml
    ├── asia-northeast1/kustomization.yml
    ├── southamerica-east1/kustomization.yml
    ├── australia-southeast1/kustomization.yml
    ├── asia-southeast1/kustomization.yml
    ├── asia-northeast3/kustomization.yml
    └── asia-south1/kustomization.yml

Each per-region/{region}/kustomization.yml patches exactly two things:

A. The REGION environment variable + topology.kubernetes.io/region annotation on each backend Deployment. Services use REGION to route pub/sub messages and tag metrics correctly.

B. The first 9 entries in the ExternalSecret — the ones that differ per region: Memorystore Redis connection details and Azure Cognitive Services credentials for that region's paired Azure location.

Everything else is inherited from the base.

What a per-region overlay looks like

Here's the europe-west2 overlay (GCP London / Azure uksouth):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: speakflow-prod
resources:
  - ../../prod.yml
  - ../../external-secret.yml
patches:
  # [A1] translate-api — REGION env (index 28) + topology annotation
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/28/value
        value: europe-west2
      - op: replace
        path: /spec/template/metadata/annotations/topology.kubernetes.io~1region
        value: europe-west2
    target:
      kind: Deployment
      name: translate-api

  # [A2] translate-agg — REGION env (index 7) + topology annotation
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/7/value
        value: europe-west2
      - op: replace
        path: /spec/template/metadata/annotations/topology.kubernetes.io~1region
        value: europe-west2
    target:
      kind: Deployment
      name: translate-agg

  # [B] ExternalSecret — regional Redis + Azure Cognitive Services (uksouth)
  - patch: |-
      - op: replace
        path: /spec/data/0/remoteRef/key
        value: speakflow-prod-europe-west2-redis-url
      - op: replace
        path: /spec/data/1/remoteRef/key
        value: speakflow-prod-europe-west2-redis-pubsub-host
      - op: replace
        path: /spec/data/2/remoteRef/key
        value: speakflow-prod-europe-west2-redis-password
      - op: replace
        path: /spec/data/3/remoteRef/key
        value: speakflow-prod-europe-west2-stt-region
      - op: replace
        path: /spec/data/4/remoteRef/key
        value: speakflow-prod-europe-west2-stt-key
      - op: replace
        path: /spec/data/5/remoteRef/key
        value: speakflow-prod-europe-west2-translator-key
      - op: replace
        path: /spec/data/6/remoteRef/key
        value: speakflow-prod-europe-west2-translator-region
      - op: replace
        path: /spec/data/7/remoteRef/key
        value: speakflow-prod-europe-west2-language-endpoint
      - op: replace
        path: /spec/data/8/remoteRef/key
        value: speakflow-prod-europe-west2-language-key
    target:
      kind: ExternalSecret
      name: translate-api

The overlay is a pure diff. It contains no credentials — only the names of secrets in GCP Secret Manager.

The index stability trap you must not ignore

JSON Patch addresses array elements by position. env/28 means the 29th environment variable in the container spec. This is load-bearing.

Deployment	REGION env index
translate-api	env[28]
translate-agg	env[7]
translate-worker	env[12]
manager-api	env[17]

If you add a new environment variable before the REGION entry in prod.yml, every overlay patch breaks silently. The REGION value gets applied to the wrong variable, and your services start routing pub/sub messages to the wrong region — a failure mode that looks like an application bug, not an infrastructure one.

Rules to live by:

Always append new env vars after REGION, never insert before it.
Do not reorder the first 9 entries in the ExternalSecret.
If you must change the order, update all 10 overlay patches before merging.

Document the indices in your README and treat that table as load-bearing infrastructure.

Secrets Management: External Secrets Operator + GCP Secret Manager

Hardcoded secrets in manifests are a non-starter for a multi-region setup. We use the External Secrets Operator (ESO) with GCP Workload Identity to sync secrets from GCP Secret Manager into Kubernetes Secrets at runtime.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: translate-api
  namespace: speakflow-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: ClusterSecretStore
  target:
    name: translate-api
    creationPolicy: Owner
  data:
    # ── Regional secrets (indices 0–8) — patched per region by Kustomize ────
    - secretKey: redisUrl              # index 0
      remoteRef:
        key: speakflow-prod-us-central1-redis-url   # ← patched by overlay
    - secretKey: redisPubsubHost       # index 1
      remoteRef:
        key: speakflow-prod-us-central1-redis-pubsub-host
    - secretKey: redisPassword         # index 2
      remoteRef:
        key: speakflow-prod-us-central1-redis-password
    - secretKey: sttRegion             # index 3
      remoteRef:
        key: speakflow-prod-us-central1-stt-region
    - secretKey: sttKey                # index 4
      remoteRef:
        key: speakflow-prod-us-central1-stt-key
    - secretKey: translatorKey         # index 5
      remoteRef:
        key: speakflow-prod-us-central1-translator-key
    - secretKey: translatorRegion      # index 6
      remoteRef:
        key: speakflow-prod-us-central1-translator-region
    - secretKey: languageEndpoint      # index 7
      remoteRef:
        key: speakflow-prod-us-central1-language-endpoint
    - secretKey: languageKey           # index 8
      remoteRef:
        key: speakflow-prod-us-central1-language-key

    # ── Shared secrets (index 9+) — same value across all regions ───────────
    - secretKey: jwtSecret             # index 9
      remoteRef:
        key: speakflow-prod-jwt-secret
    - secretKey: dbUri                 # index 10
      remoteRef:
        key: speakflow-prod-db-uri
    # ... additional shared secrets

The design splits secrets into two groups:

Regional (indices 0–8): Memorystore Redis private IP addresses, and Azure Cognitive Services API keys and region names. These are unique to each GCP+Azure region pair, provisioned in Secret Manager under a speakflow-prod-{region}-{key} naming convention.

Shared (index 9+): JWT secret, database URI, AI provider keys, email service, auth provider. Provisioned once, read by all 10 regions.

The ClusterSecretStore named gcp-secret-store is backed by GCP Workload Identity — provisioned by Terraform, not by these manifests.

Rotating a regional secret

# 1. Update the value in GCP Secret Manager (via console or gcloud)

# 2. Force an immediate ESO sync without waiting for refreshInterval
kubectl annotate externalsecret translate-api -n speakflow-prod \
  force-sync=$(date +%s) --overwrite

# 3. Pods read secrets from env on startup — rolling restart required
kubectl rollout restart deployment/translate-api -n speakflow-prod
kubectl rollout restart deployment/translate-agg -n speakflow-prod
kubectl rollout restart deployment/translate-worker -n speakflow-prod
kubectl rollout restart deployment/manager-api -n speakflow-prod

The Dual-Cloud Region Pairing

Each GCP region is paired with the geographically nearest Azure region for AI services. This is what keeps cross-cloud latency under 20ms:

GCP region	Azure Cognitive Services region
us-central1	eastus (STT) / westus2 (Translator)
europe-west2	uksouth
europe-west3 (GDPR)	germanywestcentral
northamerica-northeast1	canadacentral
asia-northeast1	japaneast
southamerica-east1	brazilsouth
australia-southeast1	australiaeast
asia-southeast1	southeastasia
asia-northeast3	koreacentral
asia-south1	centralindia

The asymmetry in us-central1 (different Azure regions for STT vs. Translator) is a real-world quirk — not every Azure service has the same regional availability. This asymmetry lives in the Secret Manager values, not in the manifests. The per-region secret speakflow-prod-us-central1-stt-region stores eastus and speakflow-prod-us-central1-translator-region stores westus2. The manifest is unaware of the difference.

Multi-Cluster Routing: GKE Fleet + ServiceExport/Import

The Gateway layer is where the multi-cluster routing happens. All 10 regional clusters are enrolled in a GKE Fleet, federated into a single logical mesh. The GKE multi-cluster Gateway controller (gke-l7-global-external-managed-mc) exposes services across it.

The key primitives are ServiceExport and ServiceImport.

In prod.yml, each service that needs to be reachable cross-cluster gets a ServiceExport:

apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: translate-api
  namespace: speakflow-prod
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: manager-api
  namespace: speakflow-prod
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: dashboard-app
  namespace: speakflow-prod

The Fleet control plane creates corresponding ServiceImport resources cluster-wide automatically.

The Gateway and HTTPRoutes are deployed once — not per region. The gke-l7-global-external-managed-mc GatewayClass handles global traffic distribution:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: speakflow-prod-gateway
  namespace: speakflow-prod
  annotations:
    networking.gke.io/certmap: "speakflow-cert-map"
spec:
  gatewayClassName: gke-l7-global-external-managed-mc
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTPS
      port: 443
  addresses:
    - type: NamedAddress
      value: speakflow-prod-static-ip

HTTPRoutes reference services by ServiceImport — this is what tells the load balancer to route to the nearest healthy backend across all clusters:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: translate-api-route
  namespace: speakflow-prod
spec:
  parentRefs:
    - name: speakflow-prod-gateway
      sectionName: https
  hostnames:
    - "api.speakflow.io"
  rules:
    # REST API traffic
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: translate-api
          kind: ServiceImport      # ← multi-cluster aware
          group: net.gke.io
          port: 80
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /

    # WebSocket traffic (Socket.IO)
    - matches:
        - path:
            type: PathPrefix
            value: /socket.io
      backendRefs:
        - name: translate-api
          kind: ServiceImport
          group: net.gke.io
          port: 80

    # Manager API
    - matches:
        - path:
            type: PathPrefix
            value: /manager
      backendRefs:
        - name: manager-api
          kind: ServiceImport
          group: net.gke.io
          port: 80

The GKE load balancer routes each request to the cluster with the lowest latency healthy backends — no application-level routing logic needed. The user in Tokyo hits asia-northeast1. The user in São Paulo hits southamerica-east1.

One thing that surprised us: GKE's managed multi-cluster Gateway handles WebSocket connection persistence correctly without extra configuration. We initially expected to write sticky-session logic ourselves; the -mc GatewayClass handled it.

Health Probes: The Live/Ready Split That Matters

One decision that significantly improved our production stability was enforcing a strict separation between liveness and readiness probe behavior.

readinessProbe:
  httpGet:
    path: /health/ready   # checks Redis + DB connectivity
    port: 3000
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /health/live    # in-process only — event loop, heap
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 15
  failureThreshold: 3

The principle: liveness must never fail due to a dependency outage.

If the managed Redis instance goes down, we don't want Kubernetes restarting all pods in a cascade — the pods are healthy, the dependency is the problem. GET /health/live checks only in-process state: is the Node.js event loop responsive, is heap usage within bounds.

GET /health/ready checks Redis and the database. When those fail, the pod is removed from the load balancer — traffic is redirected to other regions or healthy pods — but the pod is not killed. Recovery is automatic when the dependency comes back.

We also configure heap size explicitly:

env:
  - name: NODE_OPTIONS
    value: '--max-old-space-size=896'

Without this, the healthcheck library derives its heap threshold from the V8 default (1200MB), not from the container's memory limit (1024Mi). Setting it to ~87% of the limit prevents false health alerts when the container approaches its ceiling.

The SPA Init Container Pattern

The dashboard-app is a React SPA built at compile time — no runtime environment access. But we need different API endpoints, auth server URLs, and feature flag keys per environment. An init container solves this by performing token substitution on the compiled JS bundles at pod startup:

initContainers:
  - name: env-substitution
    image: us-central1-docker.pkg.dev/speakflow-prod/app/dashboard:latest
    securityContext:
      runAsUser: 0  # root required to read image files and write to emptyDir
    command:
      - /bin/sh
      - -c
      - |
        set -e
        cp -rp /usr/share/nginx/html/assets/. /mnt/assets/
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__API_URL_PLACEHOLDER__,'"$API_URL"',g' {} \;
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__AUTH_CLIENT_ID_PLACEHOLDER__,'"$AUTH_CLIENT_ID"',g' {} \;
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__FEATURE_FLAGS_KEY_PLACEHOLDER__,'"$FF_KEY"',g' {} \;
        chmod -R a+r /mnt/assets
    volumeMounts:
      - name: html-assets
        mountPath: /mnt/assets
    env:
      - name: API_URL
        value: https://api.speakflow.io
      - name: AUTH_CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: translate-api
            key: authClientId
      - name: FF_KEY
        valueFrom:
          secretKeyRef:
            name: translate-api
            key: featureFlagsKey
volumes:
  - name: html-assets
    emptyDir: {}

containers:
  - name: dashboard-app
    image: us-central1-docker.pkg.dev/speakflow-prod/app/dashboard:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 10002
      allowPrivilegeEscalation: false
      capabilities:
        drop: ['ALL']
    volumeMounts:
      - name: html-assets
        mountPath: /usr/share/nginx/html/assets

The init container copies the static assets to an emptyDir volume shared with the main container, substitutes placeholders with values from Kubernetes Secrets, then exits. nginx serves the patched files. One Docker image works in any environment without rebuilding.

The runAsUser: 0 on the init container is intentional — it needs root to read image-owned files and write to the shared volume. The main container runs with full non-root restrictions. Document this explicitly; it will come up in security reviews.

HPA + Cost-Aware Replica Scaling

The base manifest sets minReplicas: 2 for genuine HA, scaling up to 10 on CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: translate-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: translate-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Not every region has the same traffic volume. For regions still ramping up, per-region overlays patch minReplicas down to 1 and reduce replica counts on lower-priority services:

- patch: |-
    - op: replace
      path: /spec/minReplicas
      value: 1
  target:
    kind: HorizontalPodAutoscaler
    name: translate-api-hpa
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 1
  target:
    kind: Deployment
    name: translate-agg

High-traffic regions stay at the 2-replica baseline. The per-region patch is a cost control that you revisit as real traffic data comes in — not a permanent configuration.

Operations

Deploying to a region

# Render and inspect before applying
kubectl kustomize k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
  --load-restrictor=LoadRestrictionsNone

# Apply (ensure kubectl context is set to that cluster)
kubectl kustomize k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
  --load-restrictor=LoadRestrictionsNone \
  | kubectl apply -n speakflow-prod -f -

--load-restrictor=LoadRestrictionsNone is required because the overlay references files two levels above it. This relaxes kustomize's path check only — no effect on Kubernetes RBAC.

Verifying a region after deploy

# ExternalSecret sync status (should be Ready)
kubectl get externalsecret translate-api -n speakflow-prod

# Confirm the synced Secret exists
kubectl get secret translate-api -n speakflow-prod

# Check the REGION env var in a running pod
kubectl exec -n speakflow-prod deploy/translate-api -- printenv REGION

# Check topology annotation on pods
kubectl get pods -n speakflow-prod -l app=translate-api \
  -o jsonpath='{.items[*].metadata.annotations.topology\.kubernetes\.io/region}'

Bumping image tags across all regions

Image tags live in prod.yml. Change them once; all overlays inherit on next apply:

sed -i 's|:1.4.2|:1.5.0|g' k8s/manifests/speakflow/multiregion/prod.yml

Adding a new region

REGION=europe-north1
cp -r k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
      k8s/manifests/speakflow/multiregion/per-region/$REGION

sed -i "s/europe-west2/$REGION/g" \
  k8s/manifests/speakflow/multiregion/per-region/$REGION/kustomization.yml

Then provision the 9 regional secrets in GCP Secret Manager, render and verify, apply.

CI validation — render all regions on every PR

#!/usr/bin/env bash
set -euo pipefail

REGIONS=(
  us-central1 europe-west2 europe-west3
  northamerica-northeast1 asia-northeast1
  southamerica-east1 australia-southeast1
  asia-southeast1 asia-northeast3 asia-south1
)
BASE=k8s/manifests/speakflow/multiregion/per-region

for region in "${REGIONS[@]}"; do
  echo "Rendering $region..."
  kubectl kustomize "$BASE/$region" --load-restrictor=LoadRestrictionsNone > /dev/null
  echo "  OK"
done

echo "All regions rendered successfully."

Run this in CI on every PR that touches anything under the multiregion directory. It catches broken JSON Patch paths before they reach a cluster.

Key Takeaways

The dual-cloud pairing table is as important as the Kubernetes config. Running GCP compute and Azure AI services in the same geographic zone is the actual latency win. Getting the pairings wrong means your speech-to-text requests are still crossing oceans.

Kustomize JSON Patch indices are silent failures. When env/28 patches the wrong variable because you inserted something earlier in the list, services misbehave in ways that look like application bugs. Treat the index table in your README as load-bearing infrastructure.

ESO's refreshInterval is not the whole rotation story. Secrets sync every hour, but pods pick up new values only on restart. Build your rotation runbook around annotate-to-force-sync + rolling restart, not just the Secret Manager update.

The init container pattern for SPAs scales better than build-time env injection. One image, any environment. Substitution happens in under a second at pod startup and survives deployments without rebuilding.

gke-l7-global-external-managed-mc handles WebSocket routing. We expected to write sticky-session logic for WebSocket connections. The multi-cluster GatewayClass did the work. Try the managed path before building custom routing.

Scale replicas per region, not globally. The cost difference between minReplicas: 1 and minReplicas: 2 across 8 secondary regions adds up. Use traffic data to decide when to promote a region to full HA baseline.

If you're building something similar and have questions, the comments are open.

How has my relationship with AI evolved?

alfchee — Thu, 18 Jun 2026 23:32:21 +0000

Everyone is talking about how AI is helping junior developers write code they couldn't write before. That's true. But there's a less-told story: what happens when a senior developer picks up these tools. The leverage is completely different — and I didn't fully understand that until I was six months deep into it.

Here's how my working relationship with AI actually evolved, from tentative experiments to the workflow I rely on today.

Stage 1: Testing the Water with Copilot

I started where most developers start: GitHub Copilot inside VSCode, some time in 2025. I wasn't asking it to build features. I was using it for the mundane layer of senior work that nobody talks about — summary reports, technical emails that needed the right tone, quick estimations for sprint planning.

It was useful enough that I got curious. I started intentionally switching models to understand the differences: GPT-4, Claude Sonnet, Gemini X Pro, Kimi2.5, Minimax. Each had a different character. Some were better at structured reasoning, some at generating clean code, some at explaining tradeoffs. I started to develop a mental model of which tool for which job — not because any article told me to, but because I was running informal experiments every day.

At this point, AI was a productivity multiplier for things I already knew how to do. Useful, but not transformative yet.

Stage 2: The Planner/Worker Experiment

In January 2026, I started the Live Dubbing for Holoscan project — building a real-time speech-to-speech pipeline using NVIDIA's Holoscan SDK, a framework I had never touched before. The technical depth was significant: ASR models, translation, TTS, low-latency streaming, unfamiliar APIs.

I tried something I hadn't done before: using two AI systems with different roles simultaneously.

Gemini 2.5 Pro as the Planner. I used it for research, architecture decisions, and understanding unfamiliar territory. Give it a problem, get a structured breakdown of the approach.

GitHub Copilot + Claude Sonnet as the Worker. The implementation layer. Once I knew what to build and why, Copilot handled the implementation details inside the IDE.

This separation mattered more than I expected. The Planner role required broad context, reasoning across documents, and understanding why — things that benefit from a model with a large context window and strong reasoning. The Worker role required precision and staying in the flow of actual code. Using the same model for both is like asking the architect to also lay the bricks.

Stage 3: The Realization That Changed How I Think About This

When I got access to Claude and started using Cowork and Claude Code together, something clicked that I want to be direct about.

I work in Python on our production services. My honest Python level is junior-to-intermediate. I can read it, I understand what's happening, I can debug it — but I wouldn't have claimed the ability to build a production-grade FastAPI service with GPU-accelerated audio processing from scratch, not at the speed a project requires.

And yet: I did. The code shipped. It works in production. It handles real users.

Here's why: AI doesn't know what good architecture looks like. I do.

A junior developer using AI will generate code that works. A senior developer using AI will generate code that works and is structured correctly, handles edge cases, scales appropriately, and fits into a larger system that doesn't collapse six months later. The AI writes the lines. The experience decides what lines to write, in what order, with what constraints.

The gap between a junior and a senior developer was never just about knowing syntax. It was about judgment — knowing which problems are actually hard, which abstractions hold under load, which shortcuts will cost you later. AI doesn't shrink that gap. It amplifies whatever judgment you already have.

Stage 4: The Current Workflow

Today my working process looks like this:

Research and planning with AI first. Before I write a ticket or start a branch, I use Claude to think through the problem. I describe what I'm building, the constraints, the existing architecture. The output is a structured approach I can actually defend in a planning meeting.

Plans become visible artifacts. I take what comes out of the planning session and push it into Jira and GitHub Projects. This is important — the plan isn't just in my head or in a chat window, it lives where the team can see it, comment on it, and hold me accountable to it.

Automation removes the reporting overhead. I use n8n to automate my daily work updates. My manager gets a status report generated from what actually happened in Jira and GitHub — no manually written updates, no "what did you do this week" scrambles. This wasn't a complex implementation, but it saves me real time every week and makes my work more visible without extra effort.

MCP servers connect the tools. The part that's hardest to explain to someone who hasn't tried it: when your AI assistant can directly query your project management system, read your GitHub PRs, and execute workflows — instead of you copying and pasting between systems — the overhead drops significantly. I have MCP connections to GitHub, Jira, and n8n, and the difference in how fluid the work feels is real.

What I'd Tell Someone Starting This Journey

Don't start with "how do I use AI to write code faster." Start with "what do I actually spend time on that isn't the interesting part of my job?"

For me that was: writing status updates, translating architecture decisions into tickets, filling gaps in languages I'm not expert in, and researching unfamiliar APIs. AI is extraordinarily good at all of those things.

The interesting part of the job — deciding what to build, why, how it fits the system, what will break under load — that part still requires you. The tools just clear enough of the other noise that you can focus on it.

I started this with GitHub Copilot and some curiosity. A year later, I'm building things I couldn't have built at the same speed or with the same quality before. Not because AI replaced my judgment — because it finally gave my judgment enough room to do its job.

Why We Migrated from npm to pnpm — and It Wasn't About Speed

alfchee — Thu, 28 May 2026 01:58:59 +0000

We didn't migrate to pnpm because it's faster (it is). We migrated because npm's dependency model has become a security liability — and we felt it directly in our project.

Here's what pushed us over the edge, and exactly what we changed in our Nx monorepo to make it work.

The Problem: npm audit was lying to us

Every npm install on our project ended the same way:

8 vulnerabilities (3 moderate, 4 high, 1 critical)

The frustrating part? None of them were in packages we declared. They lived deep inside transitive dependencies — packages installed by our packages' packages. We hadn't written a single line that imported them, yet they were sitting in our node_modules, executable, and trusted.

The --legacy-peer-deps flag we'd been using to silence peer dependency warnings was another sign something was off. It's a pressure valve, not a fix.

What Wasn't The Problem

Our direct dependencies: Everything we explicitly installed was fine.
Our code: No injection vectors in the application layer.
npm itself: The registry hasn't been compromised. The problem is architectural.

Root Cause: Flat `node_modules` and Unrestricted Install Scripts

npm installs all packages — yours and every transitive dependency — into a single flat node_modules directory. This means your application code can require() any installed package, even packages you never declared a dependency on. This is called phantom dependency access, and supply chain attackers exploit it deliberately.

Worse: npm runs postinstall scripts automatically for every package it installs, including transitive ones. In 2025 alone, over 450,000 malicious npm packages were published. Attacks like the Shai-Hulud worm spread by compromising transitive dependencies and using their postinstall hooks to execute arbitrary code — targeting exactly this behavior.

Our --legacy-peer-deps flag was compounding this. It bypassed peer dependency resolution, meaning npm could install multiple conflicting versions of the same package without warning — widening the attack surface further.

The Migration: What We Actually Changed

We migrated to pnpm v11 across our Nx monorepo. Here's what the change looked like in practice.

1. The Lockfile Swap

The most visible change: we deleted package-lock.json (23,975 lines) and replaced it with pnpm-lock.yaml. Beyond the format difference, pnpm's lockfile is more tamper-resistant — it doesn't store tarball sources that can be silently overwritten, and it won't install packages listed in the lockfile that aren't declared in package.json.

- package-lock.json
+ pnpm-lock.yaml

We also updated .gitignore explicitly:

# lock files — commit pnpm-lock.yaml, ignore npm/yarn locks
package-lock.json
yarn.lock
!pnpm-lock.yaml

2. Blocking Install Scripts by Default

This is the biggest security win. pnpm v11 blocks postinstall and build scripts for all packages by default. To run scripts, packages must be explicitly allowlisted in pnpm-workspace.yaml:

# pnpm-workspace.yaml
allowBuilds:
  '@google/genai': true
  '@nestjs/core': true
  '@swc/core': true
  bcrypt: true
  esbuild: true
  nx: true
  protobufjs: true
  sqlite3: true

Every package on this list has a legitimate reason to run a build script (native bindings, code generation). Everything else is blocked. A compromised transitive dependency can no longer silently execute code on pnpm install.

3. The `.npmrc` Configuration

We added a .npmrc at the root to configure pnpm's behavior:

# Peer dependencies
auto-install-peers=true
strict-peer-dependencies=false

# Hoisting — required for NestJS + Nx compatibility
shamefully-hoist=true

# Security
audit=true
# ignore-scripts is intentionally omitted: pnpm v11 blocks all install/build
# scripts by default unless the package is listed under allowBuilds.

The shamefully-hoist=true deserves an explanation. pnpm's default behavior uses symlinked node_modules with strict isolation — each package can only access what it explicitly declares. This is the ideal security model. However, NestJS and Nx both rely on assumptions about package resolution that require hoisting. We enabled it pragmatically, with the understanding that allowBuilds still controls script execution regardless of hoisting.

4. All Dockerfiles and CI Workflows Updated

Every Dockerfile went from:

FROM node:20-bookworm AS build-stage
COPY package-lock.json ./
RUN npm ci --legacy-peer-deps

To:

FROM node:22-bookworm AS build-stage
RUN npm install -g pnpm@11.2.2
COPY pnpm-lock.yaml ./
COPY .npmrc ./
COPY pnpm-workspace.yaml ./
RUN pnpm install --frozen-lockfile

The --frozen-lockfile flag is important in CI and production builds: it refuses to install if the lockfile is out of sync with package.json, catching accidental dependency drift before it reaches production.

GitHub Actions workflows followed the same pattern, adding pnpm/action-setup@v4 before setup-node:

- uses: pnpm/action-setup@v4

- uses: actions/setup-node@v4
  with:
    node-version: 22
    cache: 'pnpm'

- name: Install dependencies
  run: pnpm install --frozen-lockfile

5. Pinning Broken Transitive Releases

During migration we discovered that @nx versions 19.8.15 had packages that were never actually published to the registry, causing install failures. pnpm's overrides in pnpm-workspace.yaml let us pin them cleanly:

overrides:
  '@nx/express': '19.8.14'
  '@nx/vue': '19.8.14'
  '@nx/next': '19.8.14'

With npm, this kind of transitive override requires npm shrinkwrap or complex workarounds. With pnpm it's three lines.

Results

Zero --legacy-peer-deps anywhere in the codebase — peer deps resolve cleanly.
Install scripts are now an explicit allowlist, not an implicit free-for-all.
pnpm audit runs on every install and CI run automatically.
Lockfile diffs are human-readable — YAML is far easier to review in PRs than JSON.
Faster installs in CI due to pnpm's content-addressable store (bonus, not the goal).

Key Takeaway

The npm supply chain threat isn't theoretical anymore. In 2025-2026 we saw real worms spreading through transitive dependencies using postinstall hooks. The flat node_modules model means a package three levels deep from something you care about can execute code on your machine and your CI runner.

pnpm v11's default script-blocking is a genuine security control, not a cosmetic preference. The migration has friction — you'll need to build an allowBuilds list carefully and tune hoisting for your framework. But the result is a dependency install process where you know exactly what's allowed to run.

Trusting npm install to be safe is no longer a reasonable default.

WebSocket Load Balancing: The Sticky Session Problem in Real-Time Services

alfchee — Tue, 12 May 2026 05:30:32 +0000

When we started scaling our real-time Speech-to-Speech (S2S) translation service beyond a single instance, we hit a wall that HTTP services don't have to worry about: WebSockets are stateful, and load balancers aren't.

Adding a second server didn't double our capacity. It broke everything. Here's why, and how we fixed it.

The Problem: Connections Dropping on Scale-Out

Our single-instance FastAPI service handled WebSocket connections beautifully. The moment we put a load balancer in front of two instances, clients started experiencing random disconnections, lost audio mid-stream, and sessions that just... vanished.

The errors on the client side were cryptic. On the server side, we'd see a connection established on instance A, and then suddenly a message arriving at instance B — which had no idea who this client was.

What Wasn't the Cause

Network issues: TCP connections were healthy. The problem reproduced even on localhost with two instances.
Timeouts: Our WebSocket keepalive pings were working. Connections weren't idle-dropping.
Application bugs: The single-instance version was solid. Same code, different behavior at scale.

Root Cause: The Stateful Nature of WebSockets

HTTP is stateless by design. A request comes in, a response goes out, done. The load balancer can send request #1 to server A and request #2 to server B — they're independent.

WebSockets are the opposite. Once the handshake completes, a persistent, bidirectional connection is established between that specific client and that specific server instance. All subsequent messages travel over that same TCP connection.

The problem breaks into two layers:

1. The Handshake Problem
The WebSocket protocol starts as an HTTP Upgrade request. Most load balancers handle this correctly, routing the initial handshake to one server. The connection is then established with that server.

2. The Reconnection Problem
This is where things silently break. When a client reconnects (after a brief network hiccup, a tab going to sleep, a mobile radio switching towers), the load balancer has no memory of which server handled the previous session. It routes the new connection to whichever server has the lowest load — which might not be the one holding the session state.

In our case, each server instance held in-memory state: the active ASR context, the translation buffer, the TTS queue, and the session audio history. When a client reconnected to the wrong instance, all of that was gone.

The Solution: Three Approaches, One Right Answer for Us

Approach 1: Sticky Sessions (IP Hash)

The simplest fix is telling the load balancer to always route the same client to the same server, based on their IP address.

In Nginx:

upstream speech_service {
    ip_hash;  # Route same IP to same upstream
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
}

server {
    location /ws/ {
        proxy_pass http://speech_service;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;  # Don't timeout long-lived connections
    }
}

The catch: IP hashing works until it doesn't. Mobile clients switch IPs constantly. Corporate networks hide thousands of users behind one NAT IP, hammering a single server. And if one server goes down, all its sticky sessions are lost anyway.

It's a band-aid, not a solution. We used it to stop the bleeding while building something better.

Approach 2: Cookie-Based Sticky Sessions

More reliable than IP hashing. The load balancer assigns a cookie on the first request, and subsequent requests carry that cookie, allowing the balancer to route them consistently.

HAProxy configuration:

backend speech_backend
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server s1 10.0.0.1:8000 check cookie s1
    server s2 10.0.0.2:8000 check cookie s2

Better, but still fundamentally tied to a single server. Any server failure means session loss, and WebSocket Upgrade requests don't always carry cookies depending on client implementation.

Approach 3: Externalized Session State (What We Actually Did)

The real fix is to stop treating session state as something that lives on a server instance. We moved all session state into Redis, turning our servers into stateless workers.

import redis.asyncio as redis
import json
from fastapi import WebSocket

class SessionStore:
    def __init__(self, redis_url: str):
        self.redis = redis.from_url(redis_url)
        self.TTL = 3600  # 1 hour session expiry

    async def save_session(self, session_id: str, data: dict):
        await self.redis.setex(
            f"session:{session_id}",
            self.TTL,
            json.dumps(data)
        )

    async def get_session(self, session_id: str) -> dict | None:
        raw = await self.redis.get(f"session:{session_id}")
        return json.loads(raw) if raw else None

    async def delete_session(self, session_id: str):
        await self.redis.delete(f"session:{session_id}")

Each WebSocket handler now looks up and persists its state through Redis, not in-memory:

@app.websocket("/ws/{session_id}")
async def websocket_endpoint(websocket: WebSocket, session_id: str):
    await websocket.accept()

    # Restore session from Redis — works on ANY server instance
    session = await store.get_session(session_id) or {
        "language_pair": "en-es",
        "audio_buffer": [],
        "segment_count": 0
    }

    try:
        while True:
            data = await websocket.receive_bytes()

            # Process audio...
            session["segment_count"] += 1

            # Persist updated state back to Redis
            await store.save_session(session_id, session)

            await websocket.send_bytes(translated_audio)

    except WebSocketDisconnect:
        # State preserved in Redis — client can reconnect to any instance
        await store.save_session(session_id, session)

With this in place, the load balancer can route reconnections anywhere. The new server instance picks up the session exactly where it left off.

The Missing Piece: Redis Pub/Sub for Audio Streaming

Externalizing session metadata was straightforward. But our audio pipeline had a trickier problem: what if a client is actively streaming audio on instance A, and a second browser tab opens a connection that lands on instance B?

Both connections need to receive the same translated audio stream. We solved this with Redis Pub/Sub:

async def stream_audio_to_client(
    websocket: WebSocket,
    session_id: str,
    redis_client
):
    pubsub = redis_client.pubsub()
    await pubsub.subscribe(f"audio:{session_id}")

    async for message in pubsub.listen():
        if message["type"] == "message":
            audio_chunk = message["data"]
            await websocket.send_bytes(audio_chunk)

async def publish_translated_audio(
    session_id: str,
    audio_chunk: bytes,
    redis_client
):
    # Any server instance can publish — all subscribers receive it
    await redis_client.publish(f"audio:{session_id}", audio_chunk)

Now any server instance that processes a translation chunk publishes to Redis, and all WebSocket connections subscribed to that session — regardless of which server they landed on — receive the audio.

Results

Scenario	Before	After
Server failover	Session lost, client must restart	Seamless reconnect, state preserved
Client reconnect on mobile	~40% session loss rate	<1% (only within Redis TTL expiry)
Horizontal scale-out	Broke sessions	Linear capacity increase
Multi-tab same session	Not supported	Works transparently

Key Takeaway

WebSocket load balancing is a distributed state problem, not a networking problem. Sticky sessions are a reasonable stopgap but they trade one failure mode for another — they just hide the statefulness instead of solving it.

Once we treated our servers as stateless workers and Redis as the source of truth for session state, we stopped thinking about which server a client was connected to entirely. That mental shift — from "connection-bound state" to "session-bound state" — is what actually makes WebSocket services horizontally scalable.

If you're running a real-time service and haven't hit this yet: you will, the moment you add that second server.

Choosing the Right Voice: A Technical Comparison of Pocket Studio Models

alfchee — Wed, 15 Apr 2026 05:24:59 +0000

When I built Pocket Studio, my goal was simple: provide high-quality Text-to-Speech (TTS) that runs locally on a CPU. But "high quality" is a multi-dimensional constraint. Are you optimizing for time-to-first-byte? Speaker fidelity across languages? Prosody naturalism on domain-specific text? Each engine in Pocket Studio makes different architectural trade-offs to answer those questions.

I integrated three distinct engines. This article breaks down the technical differences between Pocket TTS, XTTS-v2, and Qwen3-TTS so you can make an informed deployment decision.

1. Pocket TTS: The lightweight sprinter 🏃‍♂️

Pocket TTS is built on a FastSpeech2 + HiFi-GAN pipeline — a non-autoregressive architecture that predicts mel-spectrograms in parallel rather than frame-by-frame. This is the reason for its sub-80ms TTFB: there's no sequential dependency in the acoustic model, and the HiFi-GAN vocoder is computationally cheap at roughly 22 kHz output. The full model footprint sits around 50 MB, making it viable for embedded or IoT targets.

The trade-off is expressiveness. FastSpeech2 relies on a duration predictor and pitch/energy embeddings extracted at training time, which means it has no mechanism for adapting prosody at inference. You get consistent, intelligible speech — but no emotional range, no voice adaptation, and English only.

Best for: CLI tools, low-spec edge devices, rapid prototyping where latency is the hard constraint.
Pros: Near-zero latency, ~200 MB RAM, no license restrictions (MIT/Apache-2.0).
Cons: English-only, no voice cloning, limited prosodic expressiveness.

2. XTTS-v2: The multilingual powerhouse🌍

XTTS-v2 is a VITS2-based model augmented with a DVAE (Discrete Variational Autoencoder) speaker encoder. The DVAE encodes a reference audio clip — as short as 6 seconds — into a latent speaker embedding, which is then conditioned into the synthesis flow. This enables true zero-shot voice cloning without fine-tuning.

The model outputs at 24 kHz and supports 17 languages via a shared multilingual text encoder, making it the strongest option when you need cross-lingual voice consistency. The cost is size: the checkpoint is ~1.8 GB, and CPU inference runs roughly 12× slower than Pocket TTS per 100 tokens, with TTFB in the 800 ms–2 s range on a standard laptop.

One important operational note: XTTS-v2 is released under Coqui's CPML license, which prohibits commercial use above a revenue threshold. If you're shipping a product, this requires explicit review.

Best for: International apps, voice cloning from a short reference clip, content creation pipelines.
Pros: Zero-shot voice cloning (6 s reference), 17 languages, 24 kHz output, strong emotional range.
Cons: ~4 GB RAM, 800 ms+ TTFB, CPML license restrictions.

3. Qwen3-TTS: The all-rounder (my personal favorite)💎

Qwen3-TTS takes a fundamentally different architectural approach: it uses an LLM decoder backbone with flow-matching for the acoustic synthesis stage. Rather than conditioning on a fixed speaker embedding, it supports ICL (In-Context Learning) — you provide a ref_audio clip and a ref_text transcription, and the model uses in-context conditioning to adapt prosody and voice characteristics dynamically, treating TTS as a continuation problem rather than a lookup.

This is why it handles complex or idiomatic text better than VITS-based systems: the LLM backbone brings genuine language understanding to prosody decisions (emphasis, pausing, intonation) rather than relying solely on learned duration/pitch embeddings. The quantized checkpoint runs in ~6 GB RAM and produces 24 kHz output with TTFB in the 300–600 ms range — meaningfully faster than XTTS-v2 while producing more natural output.

The setup requirement is the ref_text parameter: for maximum quality, you should provide an accurate transcript of the reference audio. Without it, the model falls back to ASR-derived text, which introduces quality variance.

Best for: AI assistants, interactive applications, any use case where prosody naturalness matters.
Pros: ICL-based prosody control, multilingual, Apache-2.0 license, 300–600 ms TTFB, strong voice adaptation without fine-tuning.
Cons: Largest RAM footprint (~6 GB), requires ref_text for deterministic quality.

Technical comparison at a glance

Feature	Pocket TTS	XTTS-v2	Qwen3-TTS
Architecture	FastSpeech2 + HiFi-GAN	VITS2 + DVAE encoder	LLM decoder + flow matching
Model size	~50 MB	~1.8 GB	~3 GB (quantized)
TTFB (CPU)	< 80 ms	800 ms – 2 s	300 – 600 ms
Output sample rate	22 kHz	24 kHz	24 kHz
CPU RAM	~200 MB	~4 GB	~6 GB
Voice cloning	None	Zero-shot (6 s ref)ICL + X-Vector
Languages	English only	17 languages	Multilingual
License	MIT / Apache-2.0	CPML (restricted)	Apache-2.0
Prosody control	None	Embedding-based	ICL via `ref_text`

Which one should you deploy?

In Pocket Studio, switching between engines is a single Docker profile flag — the interfaces are unified. The decision comes down to your latency budget and fidelity requirements:

Choose Qwen3-TTS if you need natural prosody in a conversational AI context and can budget 300–600 ms TTFB and ~6 GB RAM. The ICL mechanism produces the most human-sounding output on modern hardware.
Choose XTTS-v2 if you need zero-shot voice cloning from a reference clip or require a specific non-English language, and your deployment context is compatible with CPML terms.
Choose Pocket TTS if you're targeting sub-100 ms response or running on constrained hardware where the other models simply won't fit.

Get started

All three engines are containerized and ready to pull from Docker Hub. The unified API means you can benchmark them against your own input corpus before committing.

🚀 Try them out here: https://github.com/alfchee/pocket-studio

What's your primary constraint — latency, naturalness, or language coverage? Drop it in the comments.

Pocket Studio: Bringing High-Performance Speech AI to Your CPU

alfchee — Mon, 06 Apr 2026 17:45:36 +0000

Lately, I’ve been spending a lot of time in the world of high-end GPU infrastructures, building real-time dubbing pipelines and working with massive AI SDKs like NVIDIA Riva. It’s an incredible space, but it often leaves a question hanging for many developers: "Do I always need a $2,000 GPU or a cloud subscription to build something great?"

The answer is no.

Today, I’m introducing Pocket Studio, a project born from the idea that Speech AI should be local-first, private, and accessible on consumer-grade hardware.

The Local-First Philosophy

When we move AI models to the cloud, we often trade away three critical things: Privacy, Cost, and Simplicity.

Privacy by Design: In a real-time speech application, audio data is sensitive. By keeping the inference on your local CPU, the data never leaves the container.
Predictable Costs: API calls add up. Running a containerized service on your own hardware costs exactly $0 in monthly subscriptions.
Developer Experience: With a "Docker-first" approach, you don't need to fight with driver versions or complex environments. If you have Docker, you have a Speech Lab.

Why the CPU Matters

While GPUs are the kings of training, modern quantization and optimization have made CPU-based inference remarkably viable for Text-to-Speech (TTS).

In Pocket Studio, I’ve integrated three models that represent the best of this balance:

Pocket TTS: The ultra-lightweight speed king.
XTTS-v2: The multilingual powerhouse with cloning capabilities.
Qwen3-TTS: My personal favorite. It offers a stunning balance of natural prosody and "human-like" flow without needing high-end VRAM.

The Stack Behind the Scenes

Building this wasn't just about picking models. It was about applying the lessons learned from production-scale systems:

FastAPI: Providing a robust, asynchronous interface.
Docker: Ensuring that the "works on my machine" promise actually holds true for everyone.
Streaming Architecture: Minimizing the time between the request and the first byte of audio.

Join the Journey

Pocket Studio is now stable and ready for experimentation. Whether you are building a local assistant, an accessibility tool, or just want to see what your CPU is capable of, I’d love for you to try it out.

🚀 Check out the repository here: https://github.com/alfchee/pocket-studio

I’m excited to see what the community builds when AI is truly in their own hands.

What’s your take on local-first AI? Are we moving too fast toward the cloud? Let's discuss in the comments!

The Hidden Costs of Real-Time: Latency vs Accuracy Trade-offs

alfchee — Fri, 03 Apr 2026 04:57:14 +0000

Hey developers! 🚀 After building a real-time speech-to-speech service with Nvidia Riva, I realized something crucial: real-time isn't just about speed—it's about making the right trade-offs. In this article, I want to share the hidden costs I discovered and how they shaped my engineering decisions.

The Illusion of "Real-Time"

When we say "real-time," we imagine instant responses. But in speech AI, there's always a tension between:

Latency: How fast we respond
Accuracy: How correct our output is
Stability: How consistent our service performs

Pick any two. All three? Impossible. That's the hidden cost nobody talks about.

The Buffer Size Dilemma

One of my first decisions was chunk size for audio streaming. Smaller chunks = faster response, but:

# Small chunks (64ms) - Lower latency, higher instability
audio_chunks = stream_audio(session_id, chunk_size=64)
# Processing: ~50ms per chunk
# Total latency: ~70ms

# Larger chunks (256ms) - Higher latency, more stable
audio_chunks = stream_audio(session_id, chunk_size=256)
# Processing: ~150ms per chunk  
# Total latency: ~200ms

With small chunks, I got complaints about "choppy" audio. With large chunks, users felt there was "lag." The solution? Adaptive buffering that adjusts based on network conditions.

GPU vs CPU: The Cost Calculation

I initially chose GPU because "it's faster." But the real cost analysis was more nuanced:

Factor	CPU (8 cores)	GPU (T4)
Per-session cost	$0.02/hr	$0.08/hr
Max concurrent	25	150
Latency	180ms	80ms
Dev complexity	Low	Medium

The GPU was 4x more expensive per session—but handled 6x more users with 2x better latency. For our scale, it was worth it. For yours? Run the numbers.

The Accuracy Trade-off Nobody Mentions

Here's the uncomfortable truth: faster models make more mistakes.

# Fast mode - Lower accuracy
config = RivaConfig(
    latency=50,  # ms
    word_boost=["um", "uh"],  # suppress filler words
)
# Result: "I um think it's working" → "I think it's working"

# Quality mode - Higher latency  
config = RivaConfig(
    latency=150,  # ms
    word_boost=[],
)
# Result: Full transcription with all filler words

Users prefer smooth, slightly inaccurate responses over accurate but choppy ones. The psychology of "feeling fast" often outweighs raw accuracy metrics.

What I Learned

Buffer adaptively - Static configurations fail in dynamic networks
Measure user perception, not just metrics - A 100ms response that feels smooth beats a 50ms response that glitches
Accept "good enough" - Perfect accuracy at 500ms latency loses to 95% accuracy at 100ms
Profile holistically - Single-component optimization means nothing if the pipeline still lags

The Bottom Line

Real-time systems aren't about eliminating latency—they're about managing expectations and making smart trade-offs. The best engineers don't chase impossible metrics; they design systems that prioritize the right compromises for their users.

From MVP to Production: Scaling a Speech AI Service

alfchee — Fri, 03 Apr 2026 04:53:13 +0000

Hey fellow builders! 🚀 After 7 articles diving into the technical details of my real-time speech service, I want to take a step back and share the bigger picture: how we scaled from a shaky MVP to a production-ready system handling 150+ concurrent users. This is the story that ties everything together.

The MVP Mindset (And Its Pitfalls)

Our first version worked—like most MVPs do. We had Flask handling WebSockets, Riva doing transcription, and basic TTS playback. Users could connect, speak, and get responses.

But "working" and "production-ready" are completely different categories:

# MVP code that "worked"
@app.websocket("/transcribe")
async def transcribe(websocket):
    await websocket.accept()
    while True:
        data = await websocket.receive_text()
        result = riva.transcribe(data)  # Fire and forget
        await websocket.send_text(result)

This code has:

No error handling
No reconnection logic
No resource cleanup
No monitoring

It worked in demos. It would fail spectacularly in production.

The Migration to FastAPI (Article #1 Recap)

The first major shift was moving to FastAPI. But here's what nobody tells you: the code migration was the easy part.

The hard part was rethinking everything:

# Production-ready approach
@app.websocket("/transcribe/{session_id}")
async def transcribe(websocket: WebSocket, session_id: str):
    await websocket.accept()

    try:
        session = await create_session(session_id)
        async for message in websocket.iter_text():
            result = await process_audio(message, session)
            await websocket.send_text(result)
    except WebSocketDisconnect:
        logger.info(f"Session {session_id} disconnected")
    finally:
        await cleanup_session(session_id)  # Critical!

Lesson: Async isn't just syntax—it's a different mental model for resource management.

Adding Resilience Layer by Layer (Article #7)

Error handling evolved in phases:

Phase 1: Basic try-catch

try:
    result = riva.transcribe(audio)
except Exception as e:
    logger.error(f"Error: {e}")

Phase 2: Categorized errors

try:
    result = riva.transcribe(audio)
except RivaConnectionError as e:
    emit_reconnection_event(session_id)
except AudioFormatError as e:
    send_error_message(websocket, "Invalid audio format")
except RateLimitError:
    queue_request_for_retry(session_id)

Phase 3: Graceful degradation

async def transcribe_with_fallback(audio):
    try:
        return await riva.transcribe(audio)
    except RivaConnectionError:
        logger.warning("Riva unavailable, using fallback T2")
        return await fallback_service.transcribe(audio)

Lesson: Error handling isn't a layer you add—it's a journey that evolves with your system.

The Logging Evolution (Article #14)

In the MVP, we had print("here") scattered everywhere. Production required structure:

import structlog

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

# Now we can correlate logs across services
logger = structlog.get_logger()
logger.info(
    "transcription_complete",
    session_id=session_id,
    latency_ms=latency,
    audio_chunks=chunk_count,
)

Lesson: Debugging production issues without structured logging is like trying to find a needle in a haystack—blindfolded.

Docker: From "It Works On My Machine" to Production (Article #9)

Our Dockerfile went through iterations:

# v1 - The MVP version
FROM python:3.9
COPY . /app
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

# v2 - Production version
FROM nvidia/cuda:11.8-runtime-ubuntu22.04

# Install Python and dependencies
RUN apt-get update && apt-get install -y python3.9 python3-pip
COPY requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy only needed files
COPY ./app ./app

# Run as non-root user
RUN useradd -m appuser
USER appuser

CMD ["python", "-m", "uvicorn", "app:app", "--host", "0.0.0.0"]

Lesson: Docker isn't just packaging—it's defining your production environment.

Audio Quality: The Unexpected Challenge (Article #4)

We thought audio streaming was "solved." Then users started complaining about gaps and choppiness.

The solution involved:

Buffering with sequence numbers
Fade-out/fade-in for seamless transitions
Adaptive buffering based on network conditions

class AudioBuffer:
    def __init__(self):
        self.chunks = {}
        self.sequence = 0

    def add_chunk(self, audio_data: bytes, seq: int):
        self.chunks[seq] = audio_data
        self.sequence = max(self.sequence, seq)

    def get_ordered_audio(self) -> bytes:
        ordered = b"".join(
            self.chunks[i] 
            for i in sorted(self.chunks.keys())
        )
        return ordered

Lesson: The "last 10%" of quality takes 90% of the effort.

What I Would Do Different

If I could restart this journey:

Start with FastAPI - Don't accumulate technical debt with Flask
Add structured logging from day 1 - You will need it
Design for failure - Every component will fail; plan for it
Monitor everything - If you can't measure it, you can't improve it
Document decisions - Future you will thank present you

The Takeaway

Building an MVP is fun. Scaling it to production is where the real engineering happens. The 7 articles in this series covered the technical pieces, but the overarching lesson is:

Production systems aren't built—they're evolved.

Each challenge we faced—async programming, error handling, logging, GPU integration, audio quality, Docker—added another layer of resilience. That's what makes a service "production-ready."

Thanks for following this journey! What's your scaling story?

Docker Deployment for GPU-Accelerated Services

alfchee — Thu, 05 Mar 2026 00:25:12 +0000

Containerizing standard Python web apps is easy. Containerizing Python apps that need to talk to NVIDIA GPUs, manage gRPC streams, handle WebSockets at scale, and integrate with complex monitoring stacks? That's a different beast.

In this article, I'll share how we structured our Docker deployment for a GPU-accelerated Speech-to-Speech service, moving from a fragile "works on my machine" setup to a robust production infrastructure.

The Challenge: GPU & Environment Complexity

We faced three main challenges:

Dual Deployment Modes: We needed to support both "Cloud" (Nvidia NVCF) and "Self-Hosted" (On-premise GPU) modes from the same image.
Log Management: High-volume WebSocket traffic generates massive logs. We needed structured JSON for machines (Grafana/Loki) but readable text for developers.
Process Management: Uvicorn needs careful tuning for async workloads to avoid blocking the event loop.

1. The Entrypoint Pattern

Instead of a simple CMD ["uvicorn", ...], we implemented a robust run.sh entrypoint script. This allows us to handle environment setup before the application starts.

Dockerfile:

# ... builds setup ...
COPY run.sh .
RUN chmod +x run.sh
ENV FASTAPI_ENV=production
CMD ["./run.sh", "--env", "production"]

run.sh (Simplified):

#!/bin/bash
# Smart defaults based on environment
if [ "$FASTAPI_ENV" = "production" ]; then
    LOG_CONFIG="json"
    WORKERS=${WORKERS:-4}
else
    LOG_CONFIG="human"
    WORKERS=1
fi

# Launch Uvicorn with calculated arguments
exec uvicorn app:app \
    --host 0.0.0.0 \
    --workers $WORKERS \
    --log-config $LOG_CONFIG

This pattern gives us flexibility. We can flip a single environment variable FASTAPI_ENV and the container completely reconfigures its logging strategy and worker count.

2. Docker Compose Profiles

We use Docker Compose not just for local dev, but for defining deployment "flavors". By having separate compose files sharing the same core image, we document the requirements for each mode effectively.

docker-compose.cloud.yml (Minimal, API Keys only):

services:
  riva-s2s:
    image: riva-s2s:latest
    environment:
      - RIVA_DEPLOYMENT_MODE=cloud
      - RIVA_API_KEY=${RIVA_API_KEY}

docker-compose.selfhosted.yml (Heavy, internal GPU networking):

services:
  riva-s2s:
    image: riva-s2s:latest
    environment:
      - RIVA_DEPLOYMENT_MODE=self-hosted
      - RIVA_ASR_SERVER=10.0.1.5:50051
      - RIVA_TTS_SERVER=10.0.1.6:50051
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

3. Solving the Logging Nightmare

In a real-time WebSocket service, standard logging is useless. You get thousands of "Connected" / "Disconnected" messages. We implemented structured JSON logging that Loki can ingest.

Key Trick:
In your logging_config.py, check the LOG_FORMAT env var. If it's json, switch your formatter to python-json-logger.

# Docker sees this:
{"timestamp": "2026-03-04T12:00:00", "level": "INFO", "session_id": "abc-123", "event": "audio_chunk_received", "size_bytes": 4096}

This allows us to write Grafana queries like:
sum by (session_id) (rate({app="riva-s2s"} | json | event="audio_chunk_received"[1m]))

Benefits of this Approach

Immutability: The same Docker image runs in cloud prototyping and on-prem production.
Observability: JSON logging + Loki means we can trace a single audio packet through the entire stack.
Scalability: The run.sh script makes it trivial for Kubernetes or Docker Swarm to override worker counts without rebuilding.

Conclusion

Containerizing AI services isn't just about wrapping Python in Linux. It's about exposing the necessary knobs (Environment Variables) to control the complex underlying hardware and software stack without rebuilding the container. Treat your Dockerfile and entrypoint as first-class code.

Solving Audio Gaps in Real-Time Speech Translation

alfchee — Thu, 05 Mar 2026 00:20:47 +0000

When building a real-time Speech-to-Speech (S2S) translation service, latency is usually the enemy everyone talks about. But there's a silent killer (quite literally) that can ruin the user experience just as effectively: audio gaps.

In our journey migrating from Flask to FastAPI and implementing Nvidia Riva, we encountered a persistent issue where our synthesized audio had audible stuttering—specifically, 20ms gaps of silence between chunks. Here’s how we diagnosed and fixed it, turning a robotic output into a smooth, natural conversation.

The Problem: "Machine Gun" Audio

Our pipeline looked standard:

Receive user audio (WebSocket)
Transcribe (ASR) & Translate (NMT)
Synthesize speech (TTS)
Stream audio back to the client

But the output sounded like a machine gun. Words were clear, but the flow was choppy. Opening the raw audio dump in Audacity revealed the culprit: consistent 20-50ms gaps of silence inserted between every audio chunk returned by the TTS service.

What Wasn't The Cause

Network Latency: The gaps were present even when saving to a local file.
Frontend Playback: The gaps existed in the source PCM data.
Sample Rates: 24kHz in, 24kHz out. No mismatches.

Root Cause Analysis

After deep diving into the Riva TTS behavior, we found three contributing factors:

Padding by Design: The TTS model often pads the beginning and end of synthesized audio with silence.
Imperfect Silence: This "silence" wasn't always digital zero (0x00). It often contained low-amplitude noise (0x01, 0x02), meaning our simple if sample != 0 checks failed to detect it.
Fragmented Synthesis: We were sending text to the TTS engine too aggressively (sentence by sentence or even phrase by phrase). Each request generated its own padding, compounding the issue.

The Solution: A Three-Pronged Approach

1. Aggressive Silence Trimming

We moved from a simple "zero check" to a threshold-based trim. Since the audio is 16-bit PCM, values under 100 are effectively inaudible but prevent "perfect silence" detection.

# Before: Only removed perfect zeros (ineffective)
non_zero = np.where(audio_data != 0)[0]

# After: Remove near-silence (threshold-based)
SILENCE_THRESHOLD = 100
non_silent = np.where(np.abs(audio_data) > SILENCE_THRESHOLD)[0]

if len(non_silent) > 0:
    # Keep only the audible part
    trimmed_audio = audio_data[non_silent[0] : non_silent[-1] + 1]

This immediately removed about 10-30ms of "dead air" per chunk.

2. Audio Crossfading (Windowing)

Even after trimming, stitching two audio clips together can cause a "click" if the waveform jumps instantly from one amplitude to another. We implemented a 5ms linear fade-out on every chunk.

def apply_fade_out(audio_bytes: bytes, sample_rate: int = 24000, fade_ms: int = 5) -> bytes:
    audio_data = np.frombuffer(audio_bytes, dtype=np.int16)
    num_samples = int(sample_rate * (fade_ms / 1000))

    # Linear ramp from 1.0 to 0.0
    fade_curve = np.linspace(1.0, 0.0, num_samples)

    # Apply to the very end of the chunk
    audio_data[-num_samples:] = audio_data[-num_samples:] * fade_curve

    return audio_data.tobytes()

This acts like a micro-crossfade, ensuring every chunks ends at zero amplitude, eliminating the "click" sound at boundaries.

3. Smart Text Aggregation

Finally, we stopped sending every partial sentence to the TTS engine. We increased our text buffer size to accumulate more context before requesting synthesis.

Before: Flush to TTS every 3 segments.
After: Flush to TTS every 5 segments (or on punctuation).

Fewer requests = fewer boundaries = fewer gaps.

Results

Audio Gaps: Eliminated.
Playback Smoothness: Indistinguishable from a single continuous file.
Latency Cost: Negligible (<2ms processing overhead).

Key Takeaway

In real-time audio, how you handle the bytes is just as important as the model generating them. Models are imperfect; your DSP pipeline needs to clean up the mess. Don't trust "silence" to be zero, and never stitch audio without smoothing the edges.

DEV Community: alfchee

allow_sales: false

allow_sales: false

The app, briefly, and the elephant

The sentence every tutorial ends with

Five providers

Dodo Payments

Creem

Polar

Lemon Squeezy

Paddle

Five for five

The part I want other developers to understand

The workaround I didn't take

What I built instead

One port

One stub

The bug this design already caught

Where I ended up

If you're reading this from a country like mine

5. Check whether your users can pay before checking whether you can be paid. Card penetration and recurring-charge support in your market may constrain the model more than any API does.

I Tested Three Vision Models on Catalog Images: OCR Was the Easy Part

1. Input Constraints: You Can't Upscale Missing Information

2. The Benchmark: 3 Models on Complex Layouts

The Baseline Test: Page 94 (Shared Price Fixture)

3. The Evaluator Is Part of the System Under Test

4. Operational Guardrails: Assistance Over Full Automation

Key Takeaways for Engineers

How to Run a Real-Time API Across 10 Global Regions on GKE — Without Duplicating Your Manifests

The Problem That Forces Multi-Region

The Stack

The Core Pattern: One Base Manifest, Ten Overlays

What a per-region overlay looks like

The index stability trap you must not ignore

Secrets Management: External Secrets Operator + GCP Secret Manager

Rotating a regional secret

The Dual-Cloud Region Pairing

Multi-Cluster Routing: GKE Fleet + ServiceExport/Import

Health Probes: The Live/Ready Split That Matters

The SPA Init Container Pattern

HPA + Cost-Aware Replica Scaling

Operations

Deploying to a region

Verifying a region after deploy

Bumping image tags across all regions

Adding a new region

CI validation — render all regions on every PR

Key Takeaways

How has my relationship with AI evolved?

Stage 1: Testing the Water with Copilot

Stage 2: The Planner/Worker Experiment

Stage 3: The Realization That Changed How I Think About This

Stage 4: The Current Workflow

What I'd Tell Someone Starting This Journey

Why We Migrated from npm to pnpm — and It Wasn't About Speed

The Problem: npm audit was lying to us

What Wasn't The Problem

Root Cause: Flat node_modules and Unrestricted Install Scripts

The Migration: What We Actually Changed

1. The Lockfile Swap

2. Blocking Install Scripts by Default

3. The .npmrc Configuration

4. All Dockerfiles and CI Workflows Updated

5. Pinning Broken Transitive Releases

Results

Key Takeaway

WebSocket Load Balancing: The Sticky Session Problem in Real-Time Services

The Problem: Connections Dropping on Scale-Out

What Wasn't the Cause

Root Cause: The Stateful Nature of WebSockets

The Solution: Three Approaches, One Right Answer for Us

Approach 1: Sticky Sessions (IP Hash)

Approach 2: Cookie-Based Sticky Sessions

Approach 3: Externalized Session State (What We Actually Did)

The Missing Piece: Redis Pub/Sub for Audio Streaming

Results

Key Takeaway

Choosing the Right Voice: A Technical Comparison of Pocket Studio Models

1. Pocket TTS: The lightweight sprinter 🏃‍♂️

2. XTTS-v2: The multilingual powerhouse🌍

Root Cause: Flat `node_modules` and Unrestricted Install Scripts

3. The `.npmrc` Configuration