DEV Community: alfchee

How to Run a Real-Time API Across 10 Global Regions on GKE — Without Duplicating Your Manifests

alfchee — Fri, 17 Jul 2026 05:55:55 +0000

When you're building a real-time speech translation product, latency isn't a performance metric — it's the product itself. A 300ms round-trip to process an audio chunk is the difference between a conversation that feels natural and one that feels broken. That's the constraint that pushed us to design a multi-cluster, multi-region Kubernetes setup across 10 GCP regions.

This article walks through the full architecture: every layer, the decision behind each one, and the gotchas that will bite you if you're not careful. I'll use a fictional app called SpeakFlow — a real-time speech translation SaaS — to illustrate the patterns.

The Problem That Forces Multi-Region

SpeakFlow processes real-time audio in a tight loop: speech-to-text, translation, text-to-speech, under strict latency budgets. A user speaking in Tokyo, routed to a cluster in us-central1, adds ~150ms of pure network latency before any compute happens. For a real-time product, that's a broken experience.

The solution is obvious in theory: run the stack close to users. 10 GCP regions, covering every major market — North America, Europe (including a GDPR-dedicated Germany cluster), Asia Pacific, South America, Australia.

The engineering challenge: do it without creating 10 independently drifting copies of your infrastructure. One team, one source of truth, 10 regions.

The Stack

Before going into the how, here's what gets deployed to every region:

translate-api — main API, handles REST and WebSockets (Socket.IO)
translate-agg — aggregates real-time transcription events
translate-worker — async batch processing for offline audio jobs
manager-api — admin operations and organization management
dashboard-app — the React SPA served via nginx

Each runs in its own GKE cluster per region, backed by regional Memorystore Redis and a paired Azure Cognitive Services instance (STT, Translator, Language).

The dual-cloud pattern — GCP for compute, Azure for AI services — is a deliberate decision. Azure's speech and translation APIs are best-in-class for our use case. Running them in the same geographic zone as the GKE cluster keeps cross-cloud latency under 20ms.

The Core Pattern: One Base Manifest, Ten Overlays

The most important architectural decision is using Kustomize with a strict base/overlay separation.

prod.yml is the single source of truth for all deployments, services, HPAs, and security configurations across all 10 regions. It is never applied directly to a cluster. It defines everything that is identical everywhere: image tags, replica counts, resource limits, probe paths, security contexts, and all secrets that are shared across regions.

k8s/manifests/speakflow/multiregion/
├── prod.yml                ← base — READ ONLY, never apply directly
├── external-secret.yml     ← base ExternalSecret (ESO v1)
├── gateway-prod.yml        ← GKE Gateway + HTTPRoutes (once, not per-region)
└── per-region/
    ├── us-central1/kustomization.yml
    ├── europe-west2/kustomization.yml
    ├── europe-west3/kustomization.yml
    ├── northamerica-northeast1/kustomization.yml
    ├── asia-northeast1/kustomization.yml
    ├── southamerica-east1/kustomization.yml
    ├── australia-southeast1/kustomization.yml
    ├── asia-southeast1/kustomization.yml
    ├── asia-northeast3/kustomization.yml
    └── asia-south1/kustomization.yml

Each per-region/{region}/kustomization.yml patches exactly two things:

A. The REGION environment variable + topology.kubernetes.io/region annotation on each backend Deployment. Services use REGION to route pub/sub messages and tag metrics correctly.

B. The first 9 entries in the ExternalSecret — the ones that differ per region: Memorystore Redis connection details and Azure Cognitive Services credentials for that region's paired Azure location.

Everything else is inherited from the base.

What a per-region overlay looks like

Here's the europe-west2 overlay (GCP London / Azure uksouth):

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: speakflow-prod
resources:
  - ../../prod.yml
  - ../../external-secret.yml
patches:
  # [A1] translate-api — REGION env (index 28) + topology annotation
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/28/value
        value: europe-west2
      - op: replace
        path: /spec/template/metadata/annotations/topology.kubernetes.io~1region
        value: europe-west2
    target:
      kind: Deployment
      name: translate-api

  # [A2] translate-agg — REGION env (index 7) + topology annotation
  - patch: |-
      - op: replace
        path: /spec/template/spec/containers/0/env/7/value
        value: europe-west2
      - op: replace
        path: /spec/template/metadata/annotations/topology.kubernetes.io~1region
        value: europe-west2
    target:
      kind: Deployment
      name: translate-agg

  # [B] ExternalSecret — regional Redis + Azure Cognitive Services (uksouth)
  - patch: |-
      - op: replace
        path: /spec/data/0/remoteRef/key
        value: speakflow-prod-europe-west2-redis-url
      - op: replace
        path: /spec/data/1/remoteRef/key
        value: speakflow-prod-europe-west2-redis-pubsub-host
      - op: replace
        path: /spec/data/2/remoteRef/key
        value: speakflow-prod-europe-west2-redis-password
      - op: replace
        path: /spec/data/3/remoteRef/key
        value: speakflow-prod-europe-west2-stt-region
      - op: replace
        path: /spec/data/4/remoteRef/key
        value: speakflow-prod-europe-west2-stt-key
      - op: replace
        path: /spec/data/5/remoteRef/key
        value: speakflow-prod-europe-west2-translator-key
      - op: replace
        path: /spec/data/6/remoteRef/key
        value: speakflow-prod-europe-west2-translator-region
      - op: replace
        path: /spec/data/7/remoteRef/key
        value: speakflow-prod-europe-west2-language-endpoint
      - op: replace
        path: /spec/data/8/remoteRef/key
        value: speakflow-prod-europe-west2-language-key
    target:
      kind: ExternalSecret
      name: translate-api

The overlay is a pure diff. It contains no credentials — only the names of secrets in GCP Secret Manager.

The index stability trap you must not ignore

JSON Patch addresses array elements by position. env/28 means the 29th environment variable in the container spec. This is load-bearing.

Deployment	REGION env index
translate-api	env[28]
translate-agg	env[7]
translate-worker	env[12]
manager-api	env[17]

If you add a new environment variable before the REGION entry in prod.yml, every overlay patch breaks silently. The REGION value gets applied to the wrong variable, and your services start routing pub/sub messages to the wrong region — a failure mode that looks like an application bug, not an infrastructure one.

Rules to live by:

Always append new env vars after REGION, never insert before it.
Do not reorder the first 9 entries in the ExternalSecret.
If you must change the order, update all 10 overlay patches before merging.

Document the indices in your README and treat that table as load-bearing infrastructure.

Secrets Management: External Secrets Operator + GCP Secret Manager

Hardcoded secrets in manifests are a non-starter for a multi-region setup. We use the External Secrets Operator (ESO) with GCP Workload Identity to sync secrets from GCP Secret Manager into Kubernetes Secrets at runtime.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: translate-api
  namespace: speakflow-prod
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: gcp-secret-store
    kind: ClusterSecretStore
  target:
    name: translate-api
    creationPolicy: Owner
  data:
    # ── Regional secrets (indices 0–8) — patched per region by Kustomize ────
    - secretKey: redisUrl              # index 0
      remoteRef:
        key: speakflow-prod-us-central1-redis-url   # ← patched by overlay
    - secretKey: redisPubsubHost       # index 1
      remoteRef:
        key: speakflow-prod-us-central1-redis-pubsub-host
    - secretKey: redisPassword         # index 2
      remoteRef:
        key: speakflow-prod-us-central1-redis-password
    - secretKey: sttRegion             # index 3
      remoteRef:
        key: speakflow-prod-us-central1-stt-region
    - secretKey: sttKey                # index 4
      remoteRef:
        key: speakflow-prod-us-central1-stt-key
    - secretKey: translatorKey         # index 5
      remoteRef:
        key: speakflow-prod-us-central1-translator-key
    - secretKey: translatorRegion      # index 6
      remoteRef:
        key: speakflow-prod-us-central1-translator-region
    - secretKey: languageEndpoint      # index 7
      remoteRef:
        key: speakflow-prod-us-central1-language-endpoint
    - secretKey: languageKey           # index 8
      remoteRef:
        key: speakflow-prod-us-central1-language-key

    # ── Shared secrets (index 9+) — same value across all regions ───────────
    - secretKey: jwtSecret             # index 9
      remoteRef:
        key: speakflow-prod-jwt-secret
    - secretKey: dbUri                 # index 10
      remoteRef:
        key: speakflow-prod-db-uri
    # ... additional shared secrets

The design splits secrets into two groups:

Regional (indices 0–8): Memorystore Redis private IP addresses, and Azure Cognitive Services API keys and region names. These are unique to each GCP+Azure region pair, provisioned in Secret Manager under a speakflow-prod-{region}-{key} naming convention.

Shared (index 9+): JWT secret, database URI, AI provider keys, email service, auth provider. Provisioned once, read by all 10 regions.

The ClusterSecretStore named gcp-secret-store is backed by GCP Workload Identity — provisioned by Terraform, not by these manifests.

Rotating a regional secret

# 1. Update the value in GCP Secret Manager (via console or gcloud)

# 2. Force an immediate ESO sync without waiting for refreshInterval
kubectl annotate externalsecret translate-api -n speakflow-prod \
  force-sync=$(date +%s) --overwrite

# 3. Pods read secrets from env on startup — rolling restart required
kubectl rollout restart deployment/translate-api -n speakflow-prod
kubectl rollout restart deployment/translate-agg -n speakflow-prod
kubectl rollout restart deployment/translate-worker -n speakflow-prod
kubectl rollout restart deployment/manager-api -n speakflow-prod

The Dual-Cloud Region Pairing

Each GCP region is paired with the geographically nearest Azure region for AI services. This is what keeps cross-cloud latency under 20ms:

GCP region	Azure Cognitive Services region
us-central1	eastus (STT) / westus2 (Translator)
europe-west2	uksouth
europe-west3 (GDPR)	germanywestcentral
northamerica-northeast1	canadacentral
asia-northeast1	japaneast
southamerica-east1	brazilsouth
australia-southeast1	australiaeast
asia-southeast1	southeastasia
asia-northeast3	koreacentral
asia-south1	centralindia

The asymmetry in us-central1 (different Azure regions for STT vs. Translator) is a real-world quirk — not every Azure service has the same regional availability. This asymmetry lives in the Secret Manager values, not in the manifests. The per-region secret speakflow-prod-us-central1-stt-region stores eastus and speakflow-prod-us-central1-translator-region stores westus2. The manifest is unaware of the difference.

Multi-Cluster Routing: GKE Fleet + ServiceExport/Import

The Gateway layer is where the multi-cluster routing happens. All 10 regional clusters are enrolled in a GKE Fleet, federated into a single logical mesh. The GKE multi-cluster Gateway controller (gke-l7-global-external-managed-mc) exposes services across it.

The key primitives are ServiceExport and ServiceImport.

In prod.yml, each service that needs to be reachable cross-cluster gets a ServiceExport:

apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: translate-api
  namespace: speakflow-prod
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: manager-api
  namespace: speakflow-prod
---
apiVersion: net.gke.io/v1
kind: ServiceExport
metadata:
  name: dashboard-app
  namespace: speakflow-prod

The Fleet control plane creates corresponding ServiceImport resources cluster-wide automatically.

The Gateway and HTTPRoutes are deployed once — not per region. The gke-l7-global-external-managed-mc GatewayClass handles global traffic distribution:

apiVersion: gateway.networking.k8s.io/v1
kind: Gateway
metadata:
  name: speakflow-prod-gateway
  namespace: speakflow-prod
  annotations:
    networking.gke.io/certmap: "speakflow-cert-map"
spec:
  gatewayClassName: gke-l7-global-external-managed-mc
  listeners:
    - name: http
      protocol: HTTP
      port: 80
    - name: https
      protocol: HTTPS
      port: 443
  addresses:
    - type: NamedAddress
      value: speakflow-prod-static-ip

HTTPRoutes reference services by ServiceImport — this is what tells the load balancer to route to the nearest healthy backend across all clusters:

apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: translate-api-route
  namespace: speakflow-prod
spec:
  parentRefs:
    - name: speakflow-prod-gateway
      sectionName: https
  hostnames:
    - "api.speakflow.io"
  rules:
    # REST API traffic
    - matches:
        - path:
            type: PathPrefix
            value: /v1
      backendRefs:
        - name: translate-api
          kind: ServiceImport      # ← multi-cluster aware
          group: net.gke.io
          port: 80
      filters:
        - type: URLRewrite
          urlRewrite:
            path:
              type: ReplacePrefixMatch
              replacePrefixMatch: /

    # WebSocket traffic (Socket.IO)
    - matches:
        - path:
            type: PathPrefix
            value: /socket.io
      backendRefs:
        - name: translate-api
          kind: ServiceImport
          group: net.gke.io
          port: 80

    # Manager API
    - matches:
        - path:
            type: PathPrefix
            value: /manager
      backendRefs:
        - name: manager-api
          kind: ServiceImport
          group: net.gke.io
          port: 80

The GKE load balancer routes each request to the cluster with the lowest latency healthy backends — no application-level routing logic needed. The user in Tokyo hits asia-northeast1. The user in São Paulo hits southamerica-east1.

One thing that surprised us: GKE's managed multi-cluster Gateway handles WebSocket connection persistence correctly without extra configuration. We initially expected to write sticky-session logic ourselves; the -mc GatewayClass handled it.

Health Probes: The Live/Ready Split That Matters

One decision that significantly improved our production stability was enforcing a strict separation between liveness and readiness probe behavior.

readinessProbe:
  httpGet:
    path: /health/ready   # checks Redis + DB connectivity
    port: 3000
  initialDelaySeconds: 15
  periodSeconds: 10
  failureThreshold: 3

livenessProbe:
  httpGet:
    path: /health/live    # in-process only — event loop, heap
    port: 3000
  initialDelaySeconds: 30
  periodSeconds: 15
  failureThreshold: 3

The principle: liveness must never fail due to a dependency outage.

If the managed Redis instance goes down, we don't want Kubernetes restarting all pods in a cascade — the pods are healthy, the dependency is the problem. GET /health/live checks only in-process state: is the Node.js event loop responsive, is heap usage within bounds.

GET /health/ready checks Redis and the database. When those fail, the pod is removed from the load balancer — traffic is redirected to other regions or healthy pods — but the pod is not killed. Recovery is automatic when the dependency comes back.

We also configure heap size explicitly:

env:
  - name: NODE_OPTIONS
    value: '--max-old-space-size=896'

Without this, the healthcheck library derives its heap threshold from the V8 default (1200MB), not from the container's memory limit (1024Mi). Setting it to ~87% of the limit prevents false health alerts when the container approaches its ceiling.

The SPA Init Container Pattern

The dashboard-app is a React SPA built at compile time — no runtime environment access. But we need different API endpoints, auth server URLs, and feature flag keys per environment. An init container solves this by performing token substitution on the compiled JS bundles at pod startup:

initContainers:
  - name: env-substitution
    image: us-central1-docker.pkg.dev/speakflow-prod/app/dashboard:latest
    securityContext:
      runAsUser: 0  # root required to read image files and write to emptyDir
    command:
      - /bin/sh
      - -c
      - |
        set -e
        cp -rp /usr/share/nginx/html/assets/. /mnt/assets/
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__API_URL_PLACEHOLDER__,'"$API_URL"',g' {} \;
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__AUTH_CLIENT_ID_PLACEHOLDER__,'"$AUTH_CLIENT_ID"',g' {} \;
        find /mnt/assets -name '*.js' \
          -exec sed -i 's,__FEATURE_FLAGS_KEY_PLACEHOLDER__,'"$FF_KEY"',g' {} \;
        chmod -R a+r /mnt/assets
    volumeMounts:
      - name: html-assets
        mountPath: /mnt/assets
    env:
      - name: API_URL
        value: https://api.speakflow.io
      - name: AUTH_CLIENT_ID
        valueFrom:
          secretKeyRef:
            name: translate-api
            key: authClientId
      - name: FF_KEY
        valueFrom:
          secretKeyRef:
            name: translate-api
            key: featureFlagsKey
volumes:
  - name: html-assets
    emptyDir: {}

containers:
  - name: dashboard-app
    image: us-central1-docker.pkg.dev/speakflow-prod/app/dashboard:latest
    securityContext:
      runAsNonRoot: true
      runAsUser: 10002
      allowPrivilegeEscalation: false
      capabilities:
        drop: ['ALL']
    volumeMounts:
      - name: html-assets
        mountPath: /usr/share/nginx/html/assets

The init container copies the static assets to an emptyDir volume shared with the main container, substitutes placeholders with values from Kubernetes Secrets, then exits. nginx serves the patched files. One Docker image works in any environment without rebuilding.

The runAsUser: 0 on the init container is intentional — it needs root to read image-owned files and write to the shared volume. The main container runs with full non-root restrictions. Document this explicitly; it will come up in security reviews.

HPA + Cost-Aware Replica Scaling

The base manifest sets minReplicas: 2 for genuine HA, scaling up to 10 on CPU:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: translate-api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: translate-api
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

Not every region has the same traffic volume. For regions still ramping up, per-region overlays patch minReplicas down to 1 and reduce replica counts on lower-priority services:

- patch: |-
    - op: replace
      path: /spec/minReplicas
      value: 1
  target:
    kind: HorizontalPodAutoscaler
    name: translate-api-hpa
- patch: |-
    - op: replace
      path: /spec/replicas
      value: 1
  target:
    kind: Deployment
    name: translate-agg

High-traffic regions stay at the 2-replica baseline. The per-region patch is a cost control that you revisit as real traffic data comes in — not a permanent configuration.

Operations

Deploying to a region

# Render and inspect before applying
kubectl kustomize k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
  --load-restrictor=LoadRestrictionsNone

# Apply (ensure kubectl context is set to that cluster)
kubectl kustomize k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
  --load-restrictor=LoadRestrictionsNone \
  | kubectl apply -n speakflow-prod -f -

--load-restrictor=LoadRestrictionsNone is required because the overlay references files two levels above it. This relaxes kustomize's path check only — no effect on Kubernetes RBAC.

Verifying a region after deploy

# ExternalSecret sync status (should be Ready)
kubectl get externalsecret translate-api -n speakflow-prod

# Confirm the synced Secret exists
kubectl get secret translate-api -n speakflow-prod

# Check the REGION env var in a running pod
kubectl exec -n speakflow-prod deploy/translate-api -- printenv REGION

# Check topology annotation on pods
kubectl get pods -n speakflow-prod -l app=translate-api \
  -o jsonpath='{.items[*].metadata.annotations.topology\.kubernetes\.io/region}'

Bumping image tags across all regions

Image tags live in prod.yml. Change them once; all overlays inherit on next apply:

sed -i 's|:1.4.2|:1.5.0|g' k8s/manifests/speakflow/multiregion/prod.yml

Adding a new region

REGION=europe-north1
cp -r k8s/manifests/speakflow/multiregion/per-region/europe-west2 \
      k8s/manifests/speakflow/multiregion/per-region/$REGION

sed -i "s/europe-west2/$REGION/g" \
  k8s/manifests/speakflow/multiregion/per-region/$REGION/kustomization.yml

Then provision the 9 regional secrets in GCP Secret Manager, render and verify, apply.

CI validation — render all regions on every PR

#!/usr/bin/env bash
set -euo pipefail

REGIONS=(
  us-central1 europe-west2 europe-west3
  northamerica-northeast1 asia-northeast1
  southamerica-east1 australia-southeast1
  asia-southeast1 asia-northeast3 asia-south1
)
BASE=k8s/manifests/speakflow/multiregion/per-region

for region in "${REGIONS[@]}"; do
  echo "Rendering $region..."
  kubectl kustomize "$BASE/$region" --load-restrictor=LoadRestrictionsNone > /dev/null
  echo "  OK"
done

echo "All regions rendered successfully."

Run this in CI on every PR that touches anything under the multiregion directory. It catches broken JSON Patch paths before they reach a cluster.

Key Takeaways

The dual-cloud pairing table is as important as the Kubernetes config. Running GCP compute and Azure AI services in the same geographic zone is the actual latency win. Getting the pairings wrong means your speech-to-text requests are still crossing oceans.

Kustomize JSON Patch indices are silent failures. When env/28 patches the wrong variable because you inserted something earlier in the list, services misbehave in ways that look like application bugs. Treat the index table in your README as load-bearing infrastructure.

ESO's refreshInterval is not the whole rotation story. Secrets sync every hour, but pods pick up new values only on restart. Build your rotation runbook around annotate-to-force-sync + rolling restart, not just the Secret Manager update.

The init container pattern for SPAs scales better than build-time env injection. One image, any environment. Substitution happens in under a second at pod startup and survives deployments without rebuilding.

gke-l7-global-external-managed-mc handles WebSocket routing. We expected to write sticky-session logic for WebSocket connections. The multi-cluster GatewayClass did the work. Try the managed path before building custom routing.

Scale replicas per region, not globally. The cost difference between minReplicas: 1 and minReplicas: 2 across 8 secondary regions adds up. Use traffic data to decide when to promote a region to full HA baseline.

If you're building something similar and have questions, the comments are open.

How has my relationship with AI evolved?

alfchee — Thu, 18 Jun 2026 23:32:21 +0000

Everyone is talking about how AI is helping junior developers write code they couldn't write before. That's true. But there's a less-told story: what happens when a senior developer picks up these tools. The leverage is completely different — and I didn't fully understand that until I was six months deep into it.

Here's how my working relationship with AI actually evolved, from tentative experiments to the workflow I rely on today.

Stage 1: Testing the Water with Copilot

I started where most developers start: GitHub Copilot inside VSCode, some time in 2025. I wasn't asking it to build features. I was using it for the mundane layer of senior work that nobody talks about — summary reports, technical emails that needed the right tone, quick estimations for sprint planning.

It was useful enough that I got curious. I started intentionally switching models to understand the differences: GPT-4, Claude Sonnet, Gemini X Pro, Kimi2.5, Minimax. Each had a different character. Some were better at structured reasoning, some at generating clean code, some at explaining tradeoffs. I started to develop a mental model of which tool for which job — not because any article told me to, but because I was running informal experiments every day.

At this point, AI was a productivity multiplier for things I already knew how to do. Useful, but not transformative yet.

Stage 2: The Planner/Worker Experiment

In January 2026, I started the Live Dubbing for Holoscan project — building a real-time speech-to-speech pipeline using NVIDIA's Holoscan SDK, a framework I had never touched before. The technical depth was significant: ASR models, translation, TTS, low-latency streaming, unfamiliar APIs.

I tried something I hadn't done before: using two AI systems with different roles simultaneously.

Gemini 2.5 Pro as the Planner. I used it for research, architecture decisions, and understanding unfamiliar territory. Give it a problem, get a structured breakdown of the approach.

GitHub Copilot + Claude Sonnet as the Worker. The implementation layer. Once I knew what to build and why, Copilot handled the implementation details inside the IDE.

This separation mattered more than I expected. The Planner role required broad context, reasoning across documents, and understanding why — things that benefit from a model with a large context window and strong reasoning. The Worker role required precision and staying in the flow of actual code. Using the same model for both is like asking the architect to also lay the bricks.

Stage 3: The Realization That Changed How I Think About This

When I got access to Claude and started using Cowork and Claude Code together, something clicked that I want to be direct about.

I work in Python on our production services. My honest Python level is junior-to-intermediate. I can read it, I understand what's happening, I can debug it — but I wouldn't have claimed the ability to build a production-grade FastAPI service with GPU-accelerated audio processing from scratch, not at the speed a project requires.

And yet: I did. The code shipped. It works in production. It handles real users.

Here's why: AI doesn't know what good architecture looks like. I do.

A junior developer using AI will generate code that works. A senior developer using AI will generate code that works and is structured correctly, handles edge cases, scales appropriately, and fits into a larger system that doesn't collapse six months later. The AI writes the lines. The experience decides what lines to write, in what order, with what constraints.

The gap between a junior and a senior developer was never just about knowing syntax. It was about judgment — knowing which problems are actually hard, which abstractions hold under load, which shortcuts will cost you later. AI doesn't shrink that gap. It amplifies whatever judgment you already have.

Stage 4: The Current Workflow

Today my working process looks like this:

Research and planning with AI first. Before I write a ticket or start a branch, I use Claude to think through the problem. I describe what I'm building, the constraints, the existing architecture. The output is a structured approach I can actually defend in a planning meeting.

Plans become visible artifacts. I take what comes out of the planning session and push it into Jira and GitHub Projects. This is important — the plan isn't just in my head or in a chat window, it lives where the team can see it, comment on it, and hold me accountable to it.

Automation removes the reporting overhead. I use n8n to automate my daily work updates. My manager gets a status report generated from what actually happened in Jira and GitHub — no manually written updates, no "what did you do this week" scrambles. This wasn't a complex implementation, but it saves me real time every week and makes my work more visible without extra effort.

MCP servers connect the tools. The part that's hardest to explain to someone who hasn't tried it: when your AI assistant can directly query your project management system, read your GitHub PRs, and execute workflows — instead of you copying and pasting between systems — the overhead drops significantly. I have MCP connections to GitHub, Jira, and n8n, and the difference in how fluid the work feels is real.

What I'd Tell Someone Starting This Journey

Don't start with "how do I use AI to write code faster." Start with "what do I actually spend time on that isn't the interesting part of my job?"

For me that was: writing status updates, translating architecture decisions into tickets, filling gaps in languages I'm not expert in, and researching unfamiliar APIs. AI is extraordinarily good at all of those things.

The interesting part of the job — deciding what to build, why, how it fits the system, what will break under load — that part still requires you. The tools just clear enough of the other noise that you can focus on it.

I started this with GitHub Copilot and some curiosity. A year later, I'm building things I couldn't have built at the same speed or with the same quality before. Not because AI replaced my judgment — because it finally gave my judgment enough room to do its job.

Why We Migrated from npm to pnpm — and It Wasn't About Speed

alfchee — Thu, 28 May 2026 01:58:59 +0000

We didn't migrate to pnpm because it's faster (it is). We migrated because npm's dependency model has become a security liability — and we felt it directly in our project.

Here's what pushed us over the edge, and exactly what we changed in our Nx monorepo to make it work.

The Problem: npm audit was lying to us

Every npm install on our project ended the same way:

8 vulnerabilities (3 moderate, 4 high, 1 critical)

The frustrating part? None of them were in packages we declared. They lived deep inside transitive dependencies — packages installed by our packages' packages. We hadn't written a single line that imported them, yet they were sitting in our node_modules, executable, and trusted.

The --legacy-peer-deps flag we'd been using to silence peer dependency warnings was another sign something was off. It's a pressure valve, not a fix.

What Wasn't The Problem

Our direct dependencies: Everything we explicitly installed was fine.
Our code: No injection vectors in the application layer.
npm itself: The registry hasn't been compromised. The problem is architectural.

Root Cause: Flat `node_modules` and Unrestricted Install Scripts

npm installs all packages — yours and every transitive dependency — into a single flat node_modules directory. This means your application code can require() any installed package, even packages you never declared a dependency on. This is called phantom dependency access, and supply chain attackers exploit it deliberately.

Worse: npm runs postinstall scripts automatically for every package it installs, including transitive ones. In 2025 alone, over 450,000 malicious npm packages were published. Attacks like the Shai-Hulud worm spread by compromising transitive dependencies and using their postinstall hooks to execute arbitrary code — targeting exactly this behavior.

Our --legacy-peer-deps flag was compounding this. It bypassed peer dependency resolution, meaning npm could install multiple conflicting versions of the same package without warning — widening the attack surface further.

The Migration: What We Actually Changed

We migrated to pnpm v11 across our Nx monorepo. Here's what the change looked like in practice.

1. The Lockfile Swap

The most visible change: we deleted package-lock.json (23,975 lines) and replaced it with pnpm-lock.yaml. Beyond the format difference, pnpm's lockfile is more tamper-resistant — it doesn't store tarball sources that can be silently overwritten, and it won't install packages listed in the lockfile that aren't declared in package.json.

- package-lock.json
+ pnpm-lock.yaml

We also updated .gitignore explicitly:

# lock files — commit pnpm-lock.yaml, ignore npm/yarn locks
package-lock.json
yarn.lock
!pnpm-lock.yaml

2. Blocking Install Scripts by Default

This is the biggest security win. pnpm v11 blocks postinstall and build scripts for all packages by default. To run scripts, packages must be explicitly allowlisted in pnpm-workspace.yaml:

# pnpm-workspace.yaml
allowBuilds:
  '@google/genai': true
  '@nestjs/core': true
  '@swc/core': true
  bcrypt: true
  esbuild: true
  nx: true
  protobufjs: true
  sqlite3: true

Every package on this list has a legitimate reason to run a build script (native bindings, code generation). Everything else is blocked. A compromised transitive dependency can no longer silently execute code on pnpm install.

3. The `.npmrc` Configuration

We added a .npmrc at the root to configure pnpm's behavior:

# Peer dependencies
auto-install-peers=true
strict-peer-dependencies=false

# Hoisting — required for NestJS + Nx compatibility
shamefully-hoist=true

# Security
audit=true
# ignore-scripts is intentionally omitted: pnpm v11 blocks all install/build
# scripts by default unless the package is listed under allowBuilds.

The shamefully-hoist=true deserves an explanation. pnpm's default behavior uses symlinked node_modules with strict isolation — each package can only access what it explicitly declares. This is the ideal security model. However, NestJS and Nx both rely on assumptions about package resolution that require hoisting. We enabled it pragmatically, with the understanding that allowBuilds still controls script execution regardless of hoisting.

4. All Dockerfiles and CI Workflows Updated

Every Dockerfile went from:

FROM node:20-bookworm AS build-stage
COPY package-lock.json ./
RUN npm ci --legacy-peer-deps

To:

FROM node:22-bookworm AS build-stage
RUN npm install -g pnpm@11.2.2
COPY pnpm-lock.yaml ./
COPY .npmrc ./
COPY pnpm-workspace.yaml ./
RUN pnpm install --frozen-lockfile

The --frozen-lockfile flag is important in CI and production builds: it refuses to install if the lockfile is out of sync with package.json, catching accidental dependency drift before it reaches production.

GitHub Actions workflows followed the same pattern, adding pnpm/action-setup@v4 before setup-node:

- uses: pnpm/action-setup@v4

- uses: actions/setup-node@v4
  with:
    node-version: 22
    cache: 'pnpm'

- name: Install dependencies
  run: pnpm install --frozen-lockfile

5. Pinning Broken Transitive Releases

During migration we discovered that @nx versions 19.8.15 had packages that were never actually published to the registry, causing install failures. pnpm's overrides in pnpm-workspace.yaml let us pin them cleanly:

overrides:
  '@nx/express': '19.8.14'
  '@nx/vue': '19.8.14'
  '@nx/next': '19.8.14'

With npm, this kind of transitive override requires npm shrinkwrap or complex workarounds. With pnpm it's three lines.

Results

Zero --legacy-peer-deps anywhere in the codebase — peer deps resolve cleanly.
Install scripts are now an explicit allowlist, not an implicit free-for-all.
pnpm audit runs on every install and CI run automatically.
Lockfile diffs are human-readable — YAML is far easier to review in PRs than JSON.
Faster installs in CI due to pnpm's content-addressable store (bonus, not the goal).

Key Takeaway

The npm supply chain threat isn't theoretical anymore. In 2025-2026 we saw real worms spreading through transitive dependencies using postinstall hooks. The flat node_modules model means a package three levels deep from something you care about can execute code on your machine and your CI runner.

pnpm v11's default script-blocking is a genuine security control, not a cosmetic preference. The migration has friction — you'll need to build an allowBuilds list carefully and tune hoisting for your framework. But the result is a dependency install process where you know exactly what's allowed to run.

Trusting npm install to be safe is no longer a reasonable default.

WebSocket Load Balancing: The Sticky Session Problem in Real-Time Services

alfchee — Tue, 12 May 2026 05:30:32 +0000

When we started scaling our real-time Speech-to-Speech (S2S) translation service beyond a single instance, we hit a wall that HTTP services don't have to worry about: WebSockets are stateful, and load balancers aren't.

Adding a second server didn't double our capacity. It broke everything. Here's why, and how we fixed it.

The Problem: Connections Dropping on Scale-Out

Our single-instance FastAPI service handled WebSocket connections beautifully. The moment we put a load balancer in front of two instances, clients started experiencing random disconnections, lost audio mid-stream, and sessions that just... vanished.

The errors on the client side were cryptic. On the server side, we'd see a connection established on instance A, and then suddenly a message arriving at instance B — which had no idea who this client was.

What Wasn't the Cause

Network issues: TCP connections were healthy. The problem reproduced even on localhost with two instances.
Timeouts: Our WebSocket keepalive pings were working. Connections weren't idle-dropping.
Application bugs: The single-instance version was solid. Same code, different behavior at scale.

Root Cause: The Stateful Nature of WebSockets

HTTP is stateless by design. A request comes in, a response goes out, done. The load balancer can send request #1 to server A and request #2 to server B — they're independent.

WebSockets are the opposite. Once the handshake completes, a persistent, bidirectional connection is established between that specific client and that specific server instance. All subsequent messages travel over that same TCP connection.

The problem breaks into two layers:

1. The Handshake Problem
The WebSocket protocol starts as an HTTP Upgrade request. Most load balancers handle this correctly, routing the initial handshake to one server. The connection is then established with that server.

2. The Reconnection Problem
This is where things silently break. When a client reconnects (after a brief network hiccup, a tab going to sleep, a mobile radio switching towers), the load balancer has no memory of which server handled the previous session. It routes the new connection to whichever server has the lowest load — which might not be the one holding the session state.

In our case, each server instance held in-memory state: the active ASR context, the translation buffer, the TTS queue, and the session audio history. When a client reconnected to the wrong instance, all of that was gone.

The Solution: Three Approaches, One Right Answer for Us

Approach 1: Sticky Sessions (IP Hash)

The simplest fix is telling the load balancer to always route the same client to the same server, based on their IP address.

In Nginx:

upstream speech_service {
    ip_hash;  # Route same IP to same upstream
    server 10.0.0.1:8000;
    server 10.0.0.2:8000;
}

server {
    location /ws/ {
        proxy_pass http://speech_service;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_read_timeout 3600s;  # Don't timeout long-lived connections
    }
}

The catch: IP hashing works until it doesn't. Mobile clients switch IPs constantly. Corporate networks hide thousands of users behind one NAT IP, hammering a single server. And if one server goes down, all its sticky sessions are lost anyway.

It's a band-aid, not a solution. We used it to stop the bleeding while building something better.

Approach 2: Cookie-Based Sticky Sessions

More reliable than IP hashing. The load balancer assigns a cookie on the first request, and subsequent requests carry that cookie, allowing the balancer to route them consistently.

HAProxy configuration:

backend speech_backend
    balance roundrobin
    cookie SERVERID insert indirect nocache
    server s1 10.0.0.1:8000 check cookie s1
    server s2 10.0.0.2:8000 check cookie s2

Better, but still fundamentally tied to a single server. Any server failure means session loss, and WebSocket Upgrade requests don't always carry cookies depending on client implementation.

Approach 3: Externalized Session State (What We Actually Did)

The real fix is to stop treating session state as something that lives on a server instance. We moved all session state into Redis, turning our servers into stateless workers.

import redis.asyncio as redis
import json
from fastapi import WebSocket

class SessionStore:
    def __init__(self, redis_url: str):
        self.redis = redis.from_url(redis_url)
        self.TTL = 3600  # 1 hour session expiry

    async def save_session(self, session_id: str, data: dict):
        await self.redis.setex(
            f"session:{session_id}",
            self.TTL,
            json.dumps(data)
        )

    async def get_session(self, session_id: str) -> dict | None:
        raw = await self.redis.get(f"session:{session_id}")
        return json.loads(raw) if raw else None

    async def delete_session(self, session_id: str):
        await self.redis.delete(f"session:{session_id}")

Each WebSocket handler now looks up and persists its state through Redis, not in-memory:

@app.websocket("/ws/{session_id}")
async def websocket_endpoint(websocket: WebSocket, session_id: str):
    await websocket.accept()

    # Restore session from Redis — works on ANY server instance
    session = await store.get_session(session_id) or {
        "language_pair": "en-es",
        "audio_buffer": [],
        "segment_count": 0
    }

    try:
        while True:
            data = await websocket.receive_bytes()

            # Process audio...
            session["segment_count"] += 1

            # Persist updated state back to Redis
            await store.save_session(session_id, session)

            await websocket.send_bytes(translated_audio)

    except WebSocketDisconnect:
        # State preserved in Redis — client can reconnect to any instance
        await store.save_session(session_id, session)

With this in place, the load balancer can route reconnections anywhere. The new server instance picks up the session exactly where it left off.

The Missing Piece: Redis Pub/Sub for Audio Streaming

Externalizing session metadata was straightforward. But our audio pipeline had a trickier problem: what if a client is actively streaming audio on instance A, and a second browser tab opens a connection that lands on instance B?

Both connections need to receive the same translated audio stream. We solved this with Redis Pub/Sub:

async def stream_audio_to_client(
    websocket: WebSocket,
    session_id: str,
    redis_client
):
    pubsub = redis_client.pubsub()
    await pubsub.subscribe(f"audio:{session_id}")

    async for message in pubsub.listen():
        if message["type"] == "message":
            audio_chunk = message["data"]
            await websocket.send_bytes(audio_chunk)

async def publish_translated_audio(
    session_id: str,
    audio_chunk: bytes,
    redis_client
):
    # Any server instance can publish — all subscribers receive it
    await redis_client.publish(f"audio:{session_id}", audio_chunk)

Now any server instance that processes a translation chunk publishes to Redis, and all WebSocket connections subscribed to that session — regardless of which server they landed on — receive the audio.

Results

Scenario	Before	After
Server failover	Session lost, client must restart	Seamless reconnect, state preserved
Client reconnect on mobile	~40% session loss rate	<1% (only within Redis TTL expiry)
Horizontal scale-out	Broke sessions	Linear capacity increase
Multi-tab same session	Not supported	Works transparently

Key Takeaway

WebSocket load balancing is a distributed state problem, not a networking problem. Sticky sessions are a reasonable stopgap but they trade one failure mode for another — they just hide the statefulness instead of solving it.

Once we treated our servers as stateless workers and Redis as the source of truth for session state, we stopped thinking about which server a client was connected to entirely. That mental shift — from "connection-bound state" to "session-bound state" — is what actually makes WebSocket services horizontally scalable.

If you're running a real-time service and haven't hit this yet: you will, the moment you add that second server.

Choosing the Right Voice: A Technical Comparison of Pocket Studio Models

alfchee — Wed, 15 Apr 2026 05:24:59 +0000

When I built Pocket Studio, my goal was simple: provide high-quality Text-to-Speech (TTS) that runs locally on a CPU. But "high quality" is a multi-dimensional constraint. Are you optimizing for time-to-first-byte? Speaker fidelity across languages? Prosody naturalism on domain-specific text? Each engine in Pocket Studio makes different architectural trade-offs to answer those questions.

I integrated three distinct engines. This article breaks down the technical differences between Pocket TTS, XTTS-v2, and Qwen3-TTS so you can make an informed deployment decision.

1. Pocket TTS: The lightweight sprinter 🏃‍♂️

Pocket TTS is built on a FastSpeech2 + HiFi-GAN pipeline — a non-autoregressive architecture that predicts mel-spectrograms in parallel rather than frame-by-frame. This is the reason for its sub-80ms TTFB: there's no sequential dependency in the acoustic model, and the HiFi-GAN vocoder is computationally cheap at roughly 22 kHz output. The full model footprint sits around 50 MB, making it viable for embedded or IoT targets.

The trade-off is expressiveness. FastSpeech2 relies on a duration predictor and pitch/energy embeddings extracted at training time, which means it has no mechanism for adapting prosody at inference. You get consistent, intelligible speech — but no emotional range, no voice adaptation, and English only.

Best for: CLI tools, low-spec edge devices, rapid prototyping where latency is the hard constraint.
Pros: Near-zero latency, ~200 MB RAM, no license restrictions (MIT/Apache-2.0).
Cons: English-only, no voice cloning, limited prosodic expressiveness.

2. XTTS-v2: The multilingual powerhouse🌍

XTTS-v2 is a VITS2-based model augmented with a DVAE (Discrete Variational Autoencoder) speaker encoder. The DVAE encodes a reference audio clip — as short as 6 seconds — into a latent speaker embedding, which is then conditioned into the synthesis flow. This enables true zero-shot voice cloning without fine-tuning.

The model outputs at 24 kHz and supports 17 languages via a shared multilingual text encoder, making it the strongest option when you need cross-lingual voice consistency. The cost is size: the checkpoint is ~1.8 GB, and CPU inference runs roughly 12× slower than Pocket TTS per 100 tokens, with TTFB in the 800 ms–2 s range on a standard laptop.

One important operational note: XTTS-v2 is released under Coqui's CPML license, which prohibits commercial use above a revenue threshold. If you're shipping a product, this requires explicit review.

Best for: International apps, voice cloning from a short reference clip, content creation pipelines.
Pros: Zero-shot voice cloning (6 s reference), 17 languages, 24 kHz output, strong emotional range.
Cons: ~4 GB RAM, 800 ms+ TTFB, CPML license restrictions.

3. Qwen3-TTS: The all-rounder (my personal favorite)💎

Qwen3-TTS takes a fundamentally different architectural approach: it uses an LLM decoder backbone with flow-matching for the acoustic synthesis stage. Rather than conditioning on a fixed speaker embedding, it supports ICL (In-Context Learning) — you provide a ref_audio clip and a ref_text transcription, and the model uses in-context conditioning to adapt prosody and voice characteristics dynamically, treating TTS as a continuation problem rather than a lookup.

This is why it handles complex or idiomatic text better than VITS-based systems: the LLM backbone brings genuine language understanding to prosody decisions (emphasis, pausing, intonation) rather than relying solely on learned duration/pitch embeddings. The quantized checkpoint runs in ~6 GB RAM and produces 24 kHz output with TTFB in the 300–600 ms range — meaningfully faster than XTTS-v2 while producing more natural output.

The setup requirement is the ref_text parameter: for maximum quality, you should provide an accurate transcript of the reference audio. Without it, the model falls back to ASR-derived text, which introduces quality variance.

Best for: AI assistants, interactive applications, any use case where prosody naturalness matters.
Pros: ICL-based prosody control, multilingual, Apache-2.0 license, 300–600 ms TTFB, strong voice adaptation without fine-tuning.
Cons: Largest RAM footprint (~6 GB), requires ref_text for deterministic quality.

Technical comparison at a glance

Feature	Pocket TTS	XTTS-v2	Qwen3-TTS
Architecture	FastSpeech2 + HiFi-GAN	VITS2 + DVAE encoder	LLM decoder + flow matching
Model size	~50 MB	~1.8 GB	~3 GB (quantized)
TTFB (CPU)	< 80 ms	800 ms – 2 s	300 – 600 ms
Output sample rate	22 kHz	24 kHz	24 kHz
CPU RAM	~200 MB	~4 GB	~6 GB
Voice cloning	None	Zero-shot (6 s ref)ICL + X-Vector
Languages	English only	17 languages	Multilingual
License	MIT / Apache-2.0	CPML (restricted)	Apache-2.0
Prosody control	None	Embedding-based	ICL via `ref_text`

Which one should you deploy?

In Pocket Studio, switching between engines is a single Docker profile flag — the interfaces are unified. The decision comes down to your latency budget and fidelity requirements:

Choose Qwen3-TTS if you need natural prosody in a conversational AI context and can budget 300–600 ms TTFB and ~6 GB RAM. The ICL mechanism produces the most human-sounding output on modern hardware.
Choose XTTS-v2 if you need zero-shot voice cloning from a reference clip or require a specific non-English language, and your deployment context is compatible with CPML terms.
Choose Pocket TTS if you're targeting sub-100 ms response or running on constrained hardware where the other models simply won't fit.

Get started

All three engines are containerized and ready to pull from Docker Hub. The unified API means you can benchmark them against your own input corpus before committing.

🚀 Try them out here: https://github.com/alfchee/pocket-studio

What's your primary constraint — latency, naturalness, or language coverage? Drop it in the comments.

Pocket Studio: Bringing High-Performance Speech AI to Your CPU

alfchee — Mon, 06 Apr 2026 17:45:36 +0000

Lately, I’ve been spending a lot of time in the world of high-end GPU infrastructures, building real-time dubbing pipelines and working with massive AI SDKs like NVIDIA Riva. It’s an incredible space, but it often leaves a question hanging for many developers: "Do I always need a $2,000 GPU or a cloud subscription to build something great?"

The answer is no.

Today, I’m introducing Pocket Studio, a project born from the idea that Speech AI should be local-first, private, and accessible on consumer-grade hardware.

The Local-First Philosophy

When we move AI models to the cloud, we often trade away three critical things: Privacy, Cost, and Simplicity.

Privacy by Design: In a real-time speech application, audio data is sensitive. By keeping the inference on your local CPU, the data never leaves the container.
Predictable Costs: API calls add up. Running a containerized service on your own hardware costs exactly $0 in monthly subscriptions.
Developer Experience: With a "Docker-first" approach, you don't need to fight with driver versions or complex environments. If you have Docker, you have a Speech Lab.

Why the CPU Matters

While GPUs are the kings of training, modern quantization and optimization have made CPU-based inference remarkably viable for Text-to-Speech (TTS).

In Pocket Studio, I’ve integrated three models that represent the best of this balance:

Pocket TTS: The ultra-lightweight speed king.
XTTS-v2: The multilingual powerhouse with cloning capabilities.
Qwen3-TTS: My personal favorite. It offers a stunning balance of natural prosody and "human-like" flow without needing high-end VRAM.

The Stack Behind the Scenes

Building this wasn't just about picking models. It was about applying the lessons learned from production-scale systems:

FastAPI: Providing a robust, asynchronous interface.
Docker: Ensuring that the "works on my machine" promise actually holds true for everyone.
Streaming Architecture: Minimizing the time between the request and the first byte of audio.

Join the Journey

Pocket Studio is now stable and ready for experimentation. Whether you are building a local assistant, an accessibility tool, or just want to see what your CPU is capable of, I’d love for you to try it out.

🚀 Check out the repository here: https://github.com/alfchee/pocket-studio

I’m excited to see what the community builds when AI is truly in their own hands.

What’s your take on local-first AI? Are we moving too fast toward the cloud? Let's discuss in the comments!

The Hidden Costs of Real-Time: Latency vs Accuracy Trade-offs

alfchee — Fri, 03 Apr 2026 04:57:14 +0000

Hey developers! 🚀 After building a real-time speech-to-speech service with Nvidia Riva, I realized something crucial: real-time isn't just about speed—it's about making the right trade-offs. In this article, I want to share the hidden costs I discovered and how they shaped my engineering decisions.

The Illusion of "Real-Time"

When we say "real-time," we imagine instant responses. But in speech AI, there's always a tension between:

Latency: How fast we respond
Accuracy: How correct our output is
Stability: How consistent our service performs

Pick any two. All three? Impossible. That's the hidden cost nobody talks about.

The Buffer Size Dilemma

One of my first decisions was chunk size for audio streaming. Smaller chunks = faster response, but:

# Small chunks (64ms) - Lower latency, higher instability
audio_chunks = stream_audio(session_id, chunk_size=64)
# Processing: ~50ms per chunk
# Total latency: ~70ms

# Larger chunks (256ms) - Higher latency, more stable
audio_chunks = stream_audio(session_id, chunk_size=256)
# Processing: ~150ms per chunk  
# Total latency: ~200ms

With small chunks, I got complaints about "choppy" audio. With large chunks, users felt there was "lag." The solution? Adaptive buffering that adjusts based on network conditions.

GPU vs CPU: The Cost Calculation

I initially chose GPU because "it's faster." But the real cost analysis was more nuanced:

Factor	CPU (8 cores)	GPU (T4)
Per-session cost	$0.02/hr	$0.08/hr
Max concurrent	25	150
Latency	180ms	80ms
Dev complexity	Low	Medium

The GPU was 4x more expensive per session—but handled 6x more users with 2x better latency. For our scale, it was worth it. For yours? Run the numbers.

The Accuracy Trade-off Nobody Mentions

Here's the uncomfortable truth: faster models make more mistakes.

# Fast mode - Lower accuracy
config = RivaConfig(
    latency=50,  # ms
    word_boost=["um", "uh"],  # suppress filler words
)
# Result: "I um think it's working" → "I think it's working"

# Quality mode - Higher latency  
config = RivaConfig(
    latency=150,  # ms
    word_boost=[],
)
# Result: Full transcription with all filler words

Users prefer smooth, slightly inaccurate responses over accurate but choppy ones. The psychology of "feeling fast" often outweighs raw accuracy metrics.

What I Learned

Buffer adaptively - Static configurations fail in dynamic networks
Measure user perception, not just metrics - A 100ms response that feels smooth beats a 50ms response that glitches
Accept "good enough" - Perfect accuracy at 500ms latency loses to 95% accuracy at 100ms
Profile holistically - Single-component optimization means nothing if the pipeline still lags

The Bottom Line

Real-time systems aren't about eliminating latency—they're about managing expectations and making smart trade-offs. The best engineers don't chase impossible metrics; they design systems that prioritize the right compromises for their users.

From MVP to Production: Scaling a Speech AI Service

alfchee — Fri, 03 Apr 2026 04:53:13 +0000

Hey fellow builders! 🚀 After 7 articles diving into the technical details of my real-time speech service, I want to take a step back and share the bigger picture: how we scaled from a shaky MVP to a production-ready system handling 150+ concurrent users. This is the story that ties everything together.

The MVP Mindset (And Its Pitfalls)

Our first version worked—like most MVPs do. We had Flask handling WebSockets, Riva doing transcription, and basic TTS playback. Users could connect, speak, and get responses.

But "working" and "production-ready" are completely different categories:

# MVP code that "worked"
@app.websocket("/transcribe")
async def transcribe(websocket):
    await websocket.accept()
    while True:
        data = await websocket.receive_text()
        result = riva.transcribe(data)  # Fire and forget
        await websocket.send_text(result)

This code has:

No error handling
No reconnection logic
No resource cleanup
No monitoring

It worked in demos. It would fail spectacularly in production.

The Migration to FastAPI (Article #1 Recap)

The first major shift was moving to FastAPI. But here's what nobody tells you: the code migration was the easy part.

The hard part was rethinking everything:

# Production-ready approach
@app.websocket("/transcribe/{session_id}")
async def transcribe(websocket: WebSocket, session_id: str):
    await websocket.accept()

    try:
        session = await create_session(session_id)
        async for message in websocket.iter_text():
            result = await process_audio(message, session)
            await websocket.send_text(result)
    except WebSocketDisconnect:
        logger.info(f"Session {session_id} disconnected")
    finally:
        await cleanup_session(session_id)  # Critical!

Lesson: Async isn't just syntax—it's a different mental model for resource management.

Adding Resilience Layer by Layer (Article #7)

Error handling evolved in phases:

Phase 1: Basic try-catch

try:
    result = riva.transcribe(audio)
except Exception as e:
    logger.error(f"Error: {e}")

Phase 2: Categorized errors

try:
    result = riva.transcribe(audio)
except RivaConnectionError as e:
    emit_reconnection_event(session_id)
except AudioFormatError as e:
    send_error_message(websocket, "Invalid audio format")
except RateLimitError:
    queue_request_for_retry(session_id)

Phase 3: Graceful degradation

async def transcribe_with_fallback(audio):
    try:
        return await riva.transcribe(audio)
    except RivaConnectionError:
        logger.warning("Riva unavailable, using fallback T2")
        return await fallback_service.transcribe(audio)

Lesson: Error handling isn't a layer you add—it's a journey that evolves with your system.

The Logging Evolution (Article #14)

In the MVP, we had print("here") scattered everywhere. Production required structure:

import structlog

structlog.configure(
    processors=[
        structlog.contextvars.merge_contextvars,
        structlog.processors.add_log_level,
        structlog.processors.TimeStamper(fmt="iso"),
        structlog.processors.JSONRenderer(),
    ]
)

# Now we can correlate logs across services
logger = structlog.get_logger()
logger.info(
    "transcription_complete",
    session_id=session_id,
    latency_ms=latency,
    audio_chunks=chunk_count,
)

Lesson: Debugging production issues without structured logging is like trying to find a needle in a haystack—blindfolded.

Docker: From "It Works On My Machine" to Production (Article #9)

Our Dockerfile went through iterations:

# v1 - The MVP version
FROM python:3.9
COPY . /app
RUN pip install -r requirements.txt
CMD ["python", "app.py"]

# v2 - Production version
FROM nvidia/cuda:11.8-runtime-ubuntu22.04

# Install Python and dependencies
RUN apt-get update && apt-get install -y python3.9 python3-pip
COPY requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy only needed files
COPY ./app ./app

# Run as non-root user
RUN useradd -m appuser
USER appuser

CMD ["python", "-m", "uvicorn", "app:app", "--host", "0.0.0.0"]

Lesson: Docker isn't just packaging—it's defining your production environment.

Audio Quality: The Unexpected Challenge (Article #4)

We thought audio streaming was "solved." Then users started complaining about gaps and choppiness.

The solution involved:

Buffering with sequence numbers
Fade-out/fade-in for seamless transitions
Adaptive buffering based on network conditions

class AudioBuffer:
    def __init__(self):
        self.chunks = {}
        self.sequence = 0

    def add_chunk(self, audio_data: bytes, seq: int):
        self.chunks[seq] = audio_data
        self.sequence = max(self.sequence, seq)

    def get_ordered_audio(self) -> bytes:
        ordered = b"".join(
            self.chunks[i] 
            for i in sorted(self.chunks.keys())
        )
        return ordered

Lesson: The "last 10%" of quality takes 90% of the effort.

What I Would Do Different

If I could restart this journey:

Start with FastAPI - Don't accumulate technical debt with Flask
Add structured logging from day 1 - You will need it
Design for failure - Every component will fail; plan for it
Monitor everything - If you can't measure it, you can't improve it
Document decisions - Future you will thank present you

The Takeaway

Building an MVP is fun. Scaling it to production is where the real engineering happens. The 7 articles in this series covered the technical pieces, but the overarching lesson is:

Production systems aren't built—they're evolved.

Each challenge we faced—async programming, error handling, logging, GPU integration, audio quality, Docker—added another layer of resilience. That's what makes a service "production-ready."

Thanks for following this journey! What's your scaling story?

Docker Deployment for GPU-Accelerated Services

alfchee — Thu, 05 Mar 2026 00:25:12 +0000

Containerizing standard Python web apps is easy. Containerizing Python apps that need to talk to NVIDIA GPUs, manage gRPC streams, handle WebSockets at scale, and integrate with complex monitoring stacks? That's a different beast.

In this article, I'll share how we structured our Docker deployment for a GPU-accelerated Speech-to-Speech service, moving from a fragile "works on my machine" setup to a robust production infrastructure.

The Challenge: GPU & Environment Complexity

We faced three main challenges:

Dual Deployment Modes: We needed to support both "Cloud" (Nvidia NVCF) and "Self-Hosted" (On-premise GPU) modes from the same image.
Log Management: High-volume WebSocket traffic generates massive logs. We needed structured JSON for machines (Grafana/Loki) but readable text for developers.
Process Management: Uvicorn needs careful tuning for async workloads to avoid blocking the event loop.

1. The Entrypoint Pattern

Instead of a simple CMD ["uvicorn", ...], we implemented a robust run.sh entrypoint script. This allows us to handle environment setup before the application starts.

Dockerfile:

# ... builds setup ...
COPY run.sh .
RUN chmod +x run.sh
ENV FASTAPI_ENV=production
CMD ["./run.sh", "--env", "production"]

run.sh (Simplified):

#!/bin/bash
# Smart defaults based on environment
if [ "$FASTAPI_ENV" = "production" ]; then
    LOG_CONFIG="json"
    WORKERS=${WORKERS:-4}
else
    LOG_CONFIG="human"
    WORKERS=1
fi

# Launch Uvicorn with calculated arguments
exec uvicorn app:app \
    --host 0.0.0.0 \
    --workers $WORKERS \
    --log-config $LOG_CONFIG

This pattern gives us flexibility. We can flip a single environment variable FASTAPI_ENV and the container completely reconfigures its logging strategy and worker count.

2. Docker Compose Profiles

We use Docker Compose not just for local dev, but for defining deployment "flavors". By having separate compose files sharing the same core image, we document the requirements for each mode effectively.

docker-compose.cloud.yml (Minimal, API Keys only):

services:
  riva-s2s:
    image: riva-s2s:latest
    environment:
      - RIVA_DEPLOYMENT_MODE=cloud
      - RIVA_API_KEY=${RIVA_API_KEY}

docker-compose.selfhosted.yml (Heavy, internal GPU networking):

services:
  riva-s2s:
    image: riva-s2s:latest
    environment:
      - RIVA_DEPLOYMENT_MODE=self-hosted
      - RIVA_ASR_SERVER=10.0.1.5:50051
      - RIVA_TTS_SERVER=10.0.1.6:50051
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

3. Solving the Logging Nightmare

In a real-time WebSocket service, standard logging is useless. You get thousands of "Connected" / "Disconnected" messages. We implemented structured JSON logging that Loki can ingest.

Key Trick:
In your logging_config.py, check the LOG_FORMAT env var. If it's json, switch your formatter to python-json-logger.

# Docker sees this:
{"timestamp": "2026-03-04T12:00:00", "level": "INFO", "session_id": "abc-123", "event": "audio_chunk_received", "size_bytes": 4096}

This allows us to write Grafana queries like:
sum by (session_id) (rate({app="riva-s2s"} | json | event="audio_chunk_received"[1m]))

Benefits of this Approach

Immutability: The same Docker image runs in cloud prototyping and on-prem production.
Observability: JSON logging + Loki means we can trace a single audio packet through the entire stack.
Scalability: The run.sh script makes it trivial for Kubernetes or Docker Swarm to override worker counts without rebuilding.

Conclusion

Containerizing AI services isn't just about wrapping Python in Linux. It's about exposing the necessary knobs (Environment Variables) to control the complex underlying hardware and software stack without rebuilding the container. Treat your Dockerfile and entrypoint as first-class code.

Solving Audio Gaps in Real-Time Speech Translation

alfchee — Thu, 05 Mar 2026 00:20:47 +0000

When building a real-time Speech-to-Speech (S2S) translation service, latency is usually the enemy everyone talks about. But there's a silent killer (quite literally) that can ruin the user experience just as effectively: audio gaps.

In our journey migrating from Flask to FastAPI and implementing Nvidia Riva, we encountered a persistent issue where our synthesized audio had audible stuttering—specifically, 20ms gaps of silence between chunks. Here’s how we diagnosed and fixed it, turning a robotic output into a smooth, natural conversation.

The Problem: "Machine Gun" Audio

Our pipeline looked standard:

Receive user audio (WebSocket)
Transcribe (ASR) & Translate (NMT)
Synthesize speech (TTS)
Stream audio back to the client

But the output sounded like a machine gun. Words were clear, but the flow was choppy. Opening the raw audio dump in Audacity revealed the culprit: consistent 20-50ms gaps of silence inserted between every audio chunk returned by the TTS service.

What Wasn't The Cause

Network Latency: The gaps were present even when saving to a local file.
Frontend Playback: The gaps existed in the source PCM data.
Sample Rates: 24kHz in, 24kHz out. No mismatches.

Root Cause Analysis

After deep diving into the Riva TTS behavior, we found three contributing factors:

Padding by Design: The TTS model often pads the beginning and end of synthesized audio with silence.
Imperfect Silence: This "silence" wasn't always digital zero (0x00). It often contained low-amplitude noise (0x01, 0x02), meaning our simple if sample != 0 checks failed to detect it.
Fragmented Synthesis: We were sending text to the TTS engine too aggressively (sentence by sentence or even phrase by phrase). Each request generated its own padding, compounding the issue.

The Solution: A Three-Pronged Approach

1. Aggressive Silence Trimming

We moved from a simple "zero check" to a threshold-based trim. Since the audio is 16-bit PCM, values under 100 are effectively inaudible but prevent "perfect silence" detection.

# Before: Only removed perfect zeros (ineffective)
non_zero = np.where(audio_data != 0)[0]

# After: Remove near-silence (threshold-based)
SILENCE_THRESHOLD = 100
non_silent = np.where(np.abs(audio_data) > SILENCE_THRESHOLD)[0]

if len(non_silent) > 0:
    # Keep only the audible part
    trimmed_audio = audio_data[non_silent[0] : non_silent[-1] + 1]

This immediately removed about 10-30ms of "dead air" per chunk.

2. Audio Crossfading (Windowing)

Even after trimming, stitching two audio clips together can cause a "click" if the waveform jumps instantly from one amplitude to another. We implemented a 5ms linear fade-out on every chunk.

def apply_fade_out(audio_bytes: bytes, sample_rate: int = 24000, fade_ms: int = 5) -> bytes:
    audio_data = np.frombuffer(audio_bytes, dtype=np.int16)
    num_samples = int(sample_rate * (fade_ms / 1000))

    # Linear ramp from 1.0 to 0.0
    fade_curve = np.linspace(1.0, 0.0, num_samples)

    # Apply to the very end of the chunk
    audio_data[-num_samples:] = audio_data[-num_samples:] * fade_curve

    return audio_data.tobytes()

This acts like a micro-crossfade, ensuring every chunks ends at zero amplitude, eliminating the "click" sound at boundaries.

3. Smart Text Aggregation

Finally, we stopped sending every partial sentence to the TTS engine. We increased our text buffer size to accumulate more context before requesting synthesis.

Before: Flush to TTS every 3 segments.
After: Flush to TTS every 5 segments (or on punctuation).

Fewer requests = fewer boundaries = fewer gaps.

Results

Audio Gaps: Eliminated.
Playback Smoothness: Indistinguishable from a single continuous file.
Latency Cost: Negligible (<2ms processing overhead).

Key Takeaway

In real-time audio, how you handle the bytes is just as important as the model generating them. Models are imperfect; your DSP pipeline needs to clean up the mess. Don't trust "silence" to be zero, and never stitch audio without smoothing the edges.

Vue.js Composables for Speech-to-Speech Translation: Building Reactive Real-Time Audio Applications

alfchee — Mon, 16 Feb 2026 17:25:16 +0000

Introduction: Why Vue.js Composables Changed Everything

When I first set out to build a real-time speech-to-speech translation service frontend, I had already implemented a React version using hooks. "How different could Vue be?" I thought. Turns out, Vue's Composition API offered a refreshingly elegant approach to managing WebSocket connections, audio streams, and reactive state that made me rethink how I structure real-time applications.

In this article, I'll share the lessons I learned building Vue.js composables for real-time speech services, the challenges I faced, and the patterns that emerged from managing reactive WebSocket connections and audio playback. Whether you're coming from React, Angular, or are a Vue veteran, there's something here for everyone building real-time audio applications.

The Challenge: Real-Time Audio Streaming in the Browser

Before we dive into the code, let's understand what we're building:

WebSocket Connection Management: Establish and maintain a persistent connection to our FastAPI backend
Audio Capture: Access the user's microphone and stream audio chunks
Reactive State: Keep the UI in sync with connection status, transcription results, and errors
Lifecycle Management: Properly clean up resources when components unmount
Error Handling: Gracefully handle network failures, permission denials, and API errors

The beauty of Vue's Composition API is that we can encapsulate each of these concerns into reusable, testable composables.

Lesson 1: Composables Are More Than React Hooks with Different Syntax

Coming from React, I initially thought composables were just hooks with a different name. I was wrong. Here's what makes them different:

Vue Composables Have Direct Reactive References

// React Hook - needs useState
const [isConnected, setIsConnected] = useState(false);

// Vue Composable - direct reactive reference
const isConnected = ref(false);

This might seem like a small difference, but it has profound implications:

No dependency arrays: Vue's reactivity system tracks dependencies automatically
No stale closures: You're always working with the current value
Simpler mental model: Just mutate the .value property

Lifecycle is More Explicit

// React needs useEffect with cleanup
useEffect(() => {
  // Setup
  return () => {
    // Cleanup
  };
}, []);

// Vue has dedicated lifecycle hooks
onUnmounted(() => {
  // Cleanup
});

Lesson 2: Building the WebSocket Composable

Let me show you the core useTranscription composable I built. This handles all WebSocket communication with our FastAPI backend.

The Basic Structure

// composables/useTranscription.ts
import { ref, onUnmounted } from 'vue';
import { v4 as uuidv4 } from 'uuid';

export function useTranscription(apiUrl = 'ws://localhost:8000') {
  // Reactive state
  const ws = ref<WebSocket | null>(null);
  const sessionId = ref<string>('');
  const isConnected = ref(false);
  const isTranscribing = ref(false);
  const partialTranscript = ref('');
  const finalTranscript = ref('');
  const error = ref<string | null>(null);

  // Methods will go here...

  // Cleanup on unmount
  onUnmounted(() => {
    disconnect();
  });

  return {
    // State
    isConnected,
    isTranscribing,
    partialTranscript,
    finalTranscript,
    error,
    sessionId,

    // Methods
    connect,
    startTranscription,
    sendAudioChunk,
    stopTranscription,
    disconnect,
  };
}

Key Design Decision: Promise-Based Connection

One mistake I made initially was making the connection synchronous. When the UI tried to start recording immediately after connecting, the WebSocket wasn't ready. Here's how I solved it:

const connect = async (): Promise<void> => {
  return new Promise((resolve, reject) => {
    try {
      sessionId.value = uuidv4();
      const url = `${apiUrl}/transcribe/${sessionId.value}`;

      ws.value = new WebSocket(url);

      ws.value.onopen = () => {
        console.log('WebSocket connected');
      };

      ws.value.onmessage = (event) => {
        const message = JSON.parse(event.data);

        if (message.event === 'connected') {
          isConnected.value = true;
          resolve(); // ✅ Resolve when truly ready
        }
        // Handle other messages...
      };

      ws.value.onerror = (err) => {
        error.value = 'WebSocket connection error';
        reject(err);
      };

      // Timeout after 10 seconds
      setTimeout(() => {
        if (!isConnected.value) {
          reject(new Error('Connection timeout'));
        }
      }, 10000);
    } catch (err) {
      reject(err);
    }
  });
};

Why this matters: The UI can now reliably wait for the connection:

// In your component
const handleConnect = async () => {
  try {
    await connect(); // ✅ Waits for actual connection
    await startTranscription(); // ✅ Safe to start now
  } catch (err) {
    console.error('Connection failed:', err);
  }
};

Lesson 3: Reactive Message Handling

The beauty of Vue's reactivity shines when handling WebSocket messages. Here's how I structured message handling:

const handleTranscript = (message: TranscriptResult) => {
  if (message.type === 'partial') {
    partialTranscript.value = message.transcript || '';
  } else if (message.type === 'final') {
    finalTranscript.value += (message.transcript || '') + ' ';
    partialTranscript.value = ''; // Clear partial
  }
};

In the template, this just works:

<template>
  <div class="transcripts">
    <!-- Updates automatically when partialTranscript changes -->
    <p class="partial">{{ partialTranscript }}</p>

    <!-- Updates automatically when finalTranscript changes -->
    <p class="final">{{ finalTranscript }}</p>
  </div>
</template>

No manual updates, no forceUpdate, no setState callbacks. Vue's reactivity system handles everything.

Lesson 4: Audio Capture Composable

The second major composable handles microphone access. Here's where browser APIs meet Vue's reactivity:

// composables/useAudioCapture.ts
import { ref, onUnmounted } from 'vue';

export function useAudioCapture() {
  const isRecording = ref(false);
  const mediaRecorder = ref<MediaRecorder | null>(null);
  const audioStream = ref<MediaStream | null>(null);

  const startRecording = async (
    onAudioData: (data: Blob) => void
  ): Promise<void> => {
    try {
      // Request microphone access
      audioStream.value = await navigator.mediaDevices.getUserMedia({
        audio: {
          channelCount: 1, // Mono
          sampleRate: 16000, // 16kHz for speech
          echoCancellation: true,
          noiseSuppression: true,
        },
      });

      mediaRecorder.value = new MediaRecorder(audioStream.value, {
        mimeType: 'audio/webm',
        audioBitsPerSecond: 16000,
      });

      // Stream audio chunks
      mediaRecorder.value.ondataavailable = (event) => {
        if (event.data.size > 0) {
          onAudioData(event.data); // Callback to send via WebSocket
        }
      };

      // Capture in 100ms chunks for real-time streaming
      mediaRecorder.value.start(100);
      isRecording.value = true;
    } catch (err) {
      console.error('Microphone access denied:', err);
      throw err;
    }
  };

  const stopRecording = () => {
    if (mediaRecorder.value && isRecording.value) {
      mediaRecorder.value.stop();
      isRecording.value = false;
    }

    // Important: Stop all tracks to release microphone
    if (audioStream.value) {
      audioStream.value.getTracks().forEach((track) => track.stop());
      audioStream.value = null;
    }
  };

  // Automatic cleanup
  onUnmounted(() => {
    stopRecording();
  });

  return {
    isRecording,
    startRecording,
    stopRecording,
  };
}

Critical Lesson: Resource Cleanup

I learned this the hard way: failing to stop media tracks keeps the microphone active even after the component unmounts. Users see that scary "microphone is active" indicator in their browser, and they panic.

Always stop media tracks:

audioStream.value.getTracks().forEach((track) => track.stop());

Lesson 5: Bringing It Together in a Component

Here's how beautifully these composables work together:

<template>
  <div class="transcription-app">
    <div class="status">
      <span :class="{ active: isConnected }">
        {{ isConnected ? '🟢 Connected' : '🔴 Disconnected' }}
      </span>
      <span :class="{ active: isRecording }">
        {{ isRecording ? '🎤 Recording' : '⏸️ Stopped' }}
      </span>
    </div>

    <div class="controls">
      <button @click="handleStart" :disabled="isRecording">
        Start Recording
      </button>
      <button @click="handleStop" :disabled="!isRecording">
        Stop Recording
      </button>
    </div>

    <div v-if="error" class="error">⚠️ {{ error }}</div>

    <div class="transcripts">
      <p v-if="partialTranscript" class="partial">
        {{ partialTranscript }}
      </p>
      <p class="final">{{ finalTranscript }}</p>
    </div>
  </div>
</template>

<script setup lang="ts">
import { onMounted } from 'vue';
import { useTranscription } from '../composables/useTranscription';
import { useAudioCapture } from '../composables/useAudioCapture';

// Composables
const {
  isConnected,
  isTranscribing,
  partialTranscript,
  finalTranscript,
  error,
  connect,
  startTranscription,
  sendAudioChunk,
  stopTranscription,
} = useTranscription('ws://localhost:8000');

const { isRecording, startRecording, stopRecording } = useAudioCapture();

// Auto-connect on mount
onMounted(async () => {
  try {
    await connect();
  } catch (err) {
    console.error('Failed to connect:', err);
  }
});

// Start recording and transcription
const handleStart = async () => {
  try {
    await startTranscription({
      language_code: 'en-US',
      enable_automatic_punctuation: true,
    });

    await startRecording((audioBlob) => {
      sendAudioChunk(audioBlob);
    });
  } catch (err) {
    console.error('Failed to start:', err);
  }
};

// Stop everything
const handleStop = async () => {
  stopRecording();
  await stopTranscription();
};
</script>

<style scoped>
.status span.active {
  color: #00ff00;
}

.partial {
  color: #888;
  font-style: italic;
}

.final {
  color: #000;
  font-weight: bold;
}

.error {
  color: red;
  padding: 10px;
  background: #ffeeee;
}
</style>

Lesson 6: Composition API vs Options API

You might be wondering: "Why use the Composition API instead of the Options API?"

Here's what I discovered:

Composition API Wins:

Better code organization: Related logic stays together
Easier to test: Composables are just functions
TypeScript support: Inference works beautifully
Reusability: Share logic across components effortlessly

Options API was Better For:

Small, simple components: Sometimes data() and methods are clearer
Learning Vue: The structure is more explicit
Team familiarity: If your team knows Options API well

For real-time applications with complex state management, the Composition API is the clear winner.

Lesson 7: Error Handling and Edge Cases

Real-world applications need robust error handling. Here are the edge cases I encountered:

1. User Denies Microphone Permission

const handleStart = async () => {
  try {
    await startRecording(onAudioData);
  } catch (err) {
    if (err.name === 'NotAllowedError') {
      error.value = 'Microphone permission denied. Please allow access.';
    } else {
      error.value = 'Failed to start recording';
    }
  }
};

2. WebSocket Connection Lost

ws.value.onclose = (event) => {
  isConnected.value = false;
  isTranscribing.value = false;

  if (!event.wasClean) {
    error.value = 'Connection lost. Please refresh and try again.';
  }
};

3. Browser Compatibility

const checkBrowserSupport = () => {
  if (!('MediaRecorder' in window)) {
    throw new Error('MediaRecorder API not supported');
  }

  if (!('WebSocket' in window)) {
    throw new Error('WebSocket API not supported');
  }
};

Lesson 8: Performance Optimization

Real-time applications demand excellent performance. Here's what worked:

1. Debounce UI Updates for Partial Transcripts

import { ref, watch } from 'vue';
import { debounce } from 'lodash-es';

const debouncedPartial = ref('');

watch(partialTranscript, debounce((newValue) => {
  debouncedPartial.value = newValue;
}, 50));

2. Use `shallowRef` for Large Objects

import { shallowRef } from 'vue';

// Don't make every property reactive
const allTranscripts = shallowRef<TranscriptResult[]>([]);

3. Lazy Load Audio Components

<script setup lang="ts">
import { defineAsyncComponent } from 'vue';

const AudioVisualizer = defineAsyncComponent(
  () => import('./AudioVisualizer.vue')
);
</script>

Lesson 9: Testing Composables

The Composition API makes testing a joy. Here's how I test the WebSocket composable:

import { describe, it, expect, vi } from 'vitest';
import { useTranscription } from '../useTranscription';

describe('useTranscription', () => {
  it('should connect to WebSocket', async () => {
    const { connect, isConnected } = useTranscription();

    await connect();

    expect(isConnected.value).toBe(true);
  });

  it('should handle transcript messages', async () => {
    const { connect, partialTranscript } = useTranscription();

    await connect();

    // Simulate WebSocket message
    const message = {
      event: 'transcript',
      type: 'partial',
      transcript: 'Hello world',
    };

    // Emit message...

    expect(partialTranscript.value).toBe('Hello world');
  });
});

Lesson 10: What I'd Do Differently

Looking back, here's what I'd change:

1. Add Retry Logic Earlier

Instead of failing on connection errors, implement exponential backoff:

const connectWithRetry = async (maxRetries = 3) => {
  for (let i = 0; i < maxRetries; i++) {
    try {
      await connect();
      return;
    } catch (err) {
      if (i === maxRetries - 1) throw err;
      await new Promise(resolve => 
        setTimeout(resolve, Math.pow(2, i) * 1000)
      );
    }
  }
};

2. Use TypeScript from Day One

The type safety caught so many bugs during refactoring.

3. Implement Audio Queueing

For speech-to-speech translation, I should have queued audio output to prevent overlapping playback.

4. Add Connection State Machine

Managing connection states (connecting, connected, disconnecting, disconnected, error) would have simplified the logic.

Comparison: Vue Composables vs React Hooks

Having implemented both, here's my honest comparison:

Aspect	Vue Composables	React Hooks
Learning Curve	Gentler for beginners	Steeper (dependency arrays, closures)
Reactivity	Automatic, magical ✨	Manual via setState
Ref Management	Explicit `.value` access	Implicit
Lifecycle	Dedicated hooks	useEffect with deps
Code Reuse	Natural and intuitive	Requires careful hook design
TypeScript	Excellent inference	Good, but needs more annotations
Performance	Fine-grained reactivity	Virtual DOM diffing
Debugging	Vue DevTools is amazing	React DevTools is great

My verdict: For real-time applications, Vue's automatic reactivity saves significant mental overhead. React hooks are more explicit and give you more control, but require more boilerplate.

Real-World Performance Results

Here's what I achieved with this Vue implementation:

Connection establishment: ~200ms average
Audio chunk transmission: 100ms intervals, <10ms latency
UI updates: 60fps even with rapid transcript updates
Memory footprint: ~15MB for the entire app
Bundle size: 45KB (gzipped, excluding Vue)

Conclusion: The Joy of Vue Composables

Building real-time speech applications with Vue.js composables has been a revelation. The Composition API's elegant reactivity system, combined with TypeScript's type safety, creates a developer experience that's hard to beat.

Key Takeaways:

Composables promote clean separation of concerns - each composable handles one responsibility
Reactivity just works - no manual state updates needed
Lifecycle management is explicit - onUnmounted makes cleanup obvious
Promise-based patterns integrate seamlessly with async/await
Testing is straightforward - composables are just functions

If you're building real-time applications in Vue, embrace the Composition API. Your future self (and your team) will thank you.

Resources

GPU-Accelerated Speech AI: Integrating Nvidia Riva in Python

alfchee — Sat, 07 Feb 2026 18:07:38 +0000

Hey AI builders! 🚀 Today I want to share how I integrated Nvidia Riva into a real-time speech service and what I learned while optimizing a GPU-backed pipeline in Python. If you’re building low-latency ASR or S2S systems, this is one of those “level-up” integrations that will show off your engineering skills and deliver real user impact.

Why GPU Acceleration Was Non‑Negotiable

We started with CPU-based inference. It worked, but in real-time systems “it works” is not enough:

Latency ballooned during peak traffic
Concurrency hit a wall because each session hogged CPU
Scaling was expensive and still failed to feel real-time

Riva flipped the equation: more throughput, lower latency, and a cleaner streaming API. That’s when I knew this integration would be a cornerstone skill in my toolbox.

What Riva Brings to the Table

Riva is Nvidia’s GPU-accelerated speech SDK. In one platform you get:

ASR (speech-to-text)
NMT (translation)
TTS (text-to-speech)
Streaming-first architecture built for real-time apps

In short: I could finally build a pipeline that felt instant to users.

The Integration Pattern That Worked

The key is embracing gRPC streaming instead of trying to force batch logic into a realtime system.

1) Secure Auth + Service Setup

import riva.client

def build_riva_auth(api_key: str, function_id: str):
    return riva.client.Auth(
        use_ssl=True,
        uri="grpc.nvcf.nvidia.com:443",
        metadata_args=[
            ["function-id", function_id],
            ["authorization", f"Bearer {api_key}"]
        ]
    )

auth = build_riva_auth(api_key="YOUR_KEY", function_id="YOUR_FUNCTION_ID")
asr_service = riva.client.ASRService(auth)

2) Streaming Generator (The Heart of It)

import queue

class RivaStream:
    def __init__(self, asr_service):
        self.asr_service = asr_service
        self.audio_queue = queue.Queue()

    def audio_generator(self):
        while True:
            chunk = self.audio_queue.get()
            if chunk is None:
                break
            yield chunk

    def push(self, chunk: bytes):
        self.audio_queue.put(chunk)

    def close(self):
        self.audio_queue.put(None)

3) Streaming Config That Fits Real-Time

import riva.client

def build_streaming_config(lang="en"):
    return riva.client.StreamingRecognitionConfig(
        config=riva.client.RecognitionConfig(
            encoding=riva.client.AudioEncoding.LINEAR_PCM,
            sample_rate_hertz=16000,
            language_code=lang,
            max_alternatives=1,
            enable_automatic_punctuation=True,
            audio_channel_count=1,
        ),
        interim_results=True,
    )

4) Consuming the Stream

def stream_transcription(riva_stream, config):
    responses = riva_stream.asr_service.streaming_response_generator(
        audio_chunks=riva_stream.audio_generator(),
        streaming_config=config
    )

    for response in responses:
        if not response.results:
            continue
        for result in response.results:
            if result.alternatives:
                yield {
                    "text": result.alternatives[0].transcript,
                    "is_final": result.is_final,
                    "stability": result.stability
                }

That generator-based design gave me a clean, testable, and scalable integration point—exactly the kind of engineering pattern I’m proud to showcase.

The Real-Time WebSocket Bridge

I wired the Riva stream into FastAPI WebSockets to support live audio ingestion and low-latency results.

from fastapi import FastAPI, WebSocket, WebSocketDisconnect
import json
import asyncio

app = FastAPI()

@app.websocket("/transcribe/{session_id}")
async def transcribe(session_id: str, websocket: WebSocket):
    await websocket.accept()
    riva_stream = RivaStream(asr_service)
    config = build_streaming_config(lang="en")

    async def send_results():
        for result in stream_transcription(riva_stream, config):
            await websocket.send_json({
                "event": "transcription",
                **result
            })

    try:
        task = asyncio.create_task(send_results())
        while True:
            audio_chunk = await websocket.receive_bytes()
            riva_stream.push(audio_chunk)
    except WebSocketDisconnect:
        pass
    finally:
        riva_stream.close()
        await task

Reliability: Where Production Skills Really Show

This is where I invested time to make it production-grade:

Retry logic with exponential backoff for gRPC failures
Timeouts to avoid stuck streams
Graceful disconnect handling to prevent leaked resources
Structured logs for tracing latency spikes

import grpc
from tenacity import retry, stop_after_attempt, wait_exponential

class ResilientRivaClient:
    def __init__(self, api_key: str, function_id: str):
        self.api_key = api_key
        self.function_id = function_id
        self.service = self._create_service()

    @retry(stop=stop_after_attempt(3), wait=wait_exponential(multiplier=1, min=2, max=10))
    def _create_service(self):
        auth = build_riva_auth(self.api_key, self.function_id)
        return riva.client.ASRService(auth)

    def reset(self):
        self.service = self._create_service()

Performance Tuning That Actually Matters

These are the optimizations that delivered real-world gains:

Chunk sizing: ~100ms frames kept latency low without spamming gRPC.
Connection reuse: pooling channels reduced handshake costs.
Backpressure-aware queues: prevented runaway memory usage.
GPU telemetry: kept utilization in a healthy range.

CHUNK_SIZE = 3200  # 16kHz * 2 bytes * 0.1s

Results I Was Proud to Share

Latency dropped by ~90% in streaming use cases
Concurrent sessions per instance jumped 10–15x
User experience felt instant—no more “wait for it” delays

These wins weren’t just technical; they proved I could design systems that scale in real-world constraints.

Final Takeaways

Integrating Riva wasn’t just about faster inference. It was about building an architecture that’s clean, resilient, and scalable. From gRPC streaming patterns to production-grade error recovery, this project sharpened the exact skills companies look for in real-time AI engineers.

If you’re considering GPU-accelerated speech AI, start small, embrace streaming early, and invest in reliability from day one. Your users will feel the difference—and your engineering profile will too.

DEV Community: alfchee

How to Run a Real-Time API Across 10 Global Regions on GKE — Without Duplicating Your Manifests

The Problem That Forces Multi-Region

The Stack

The Core Pattern: One Base Manifest, Ten Overlays

What a per-region overlay looks like

The index stability trap you must not ignore

Secrets Management: External Secrets Operator + GCP Secret Manager

Rotating a regional secret

The Dual-Cloud Region Pairing

Multi-Cluster Routing: GKE Fleet + ServiceExport/Import

Health Probes: The Live/Ready Split That Matters

The SPA Init Container Pattern

HPA + Cost-Aware Replica Scaling

Operations

Deploying to a region

Verifying a region after deploy

Bumping image tags across all regions

Adding a new region

CI validation — render all regions on every PR

Key Takeaways

How has my relationship with AI evolved?

Stage 1: Testing the Water with Copilot

Stage 2: The Planner/Worker Experiment

Stage 3: The Realization That Changed How I Think About This

Stage 4: The Current Workflow

What I'd Tell Someone Starting This Journey

Why We Migrated from npm to pnpm — and It Wasn't About Speed

The Problem: npm audit was lying to us

What Wasn't The Problem

Root Cause: Flat node_modules and Unrestricted Install Scripts

The Migration: What We Actually Changed

1. The Lockfile Swap

2. Blocking Install Scripts by Default

3. The .npmrc Configuration

4. All Dockerfiles and CI Workflows Updated

5. Pinning Broken Transitive Releases

Results

Key Takeaway

WebSocket Load Balancing: The Sticky Session Problem in Real-Time Services

The Problem: Connections Dropping on Scale-Out

What Wasn't the Cause

Root Cause: The Stateful Nature of WebSockets

The Solution: Three Approaches, One Right Answer for Us

Approach 1: Sticky Sessions (IP Hash)

Approach 2: Cookie-Based Sticky Sessions

Approach 3: Externalized Session State (What We Actually Did)

The Missing Piece: Redis Pub/Sub for Audio Streaming

Results

Key Takeaway

Choosing the Right Voice: A Technical Comparison of Pocket Studio Models

1. Pocket TTS: The lightweight sprinter 🏃‍♂️

2. XTTS-v2: The multilingual powerhouse🌍

3. Qwen3-TTS: The all-rounder (my personal favorite)💎

Technical comparison at a glance

Which one should you deploy?

Get started

Pocket Studio: Bringing High-Performance Speech AI to Your CPU

The Local-First Philosophy

Why the CPU Matters

The Stack Behind the Scenes

Join the Journey

The Hidden Costs of Real-Time: Latency vs Accuracy Trade-offs

The Illusion of "Real-Time"

The Buffer Size Dilemma

GPU vs CPU: The Cost Calculation

The Accuracy Trade-off Nobody Mentions

What I Learned

The Bottom Line

From MVP to Production: Scaling a Speech AI Service

The MVP Mindset (And Its Pitfalls)

The Migration to FastAPI (Article #1 Recap)

Adding Resilience Layer by Layer (Article #7)

Phase 1: Basic try-catch

Phase 2: Categorized errors

Phase 3: Graceful degradation

The Logging Evolution (Article #14)

Docker: From "It Works On My Machine" to Production (Article #9)

Audio Quality: The Unexpected Challenge (Article #4)

What I Would Do Different

Root Cause: Flat `node_modules` and Unrestricted Install Scripts

3. The `.npmrc` Configuration

2. Use `shallowRef` for Large Objects