A parent's Hindi voice note hides the ask at 0:15: "send ₹5,000 before Sunday". The note is megabytes of audio nobody can search, and when a phone fills up, voice notes are what gets deleted. I wanted to keep the voice small and the ask impossible to miss, so I built Hearth for the Nebius × NVIDIA Global AI Hackathon.
The idea rests on one number I had not measured yet: how much does a 1.78 kbps neural codec hurt Hindi speech? The codec's model card has no Hindi result. So before anything else I wrote a gate: transcribe the original, encode it with NVIDIA's nemo-nano-codec-22khz-1.78kbps-12.5fps, decode it, transcribe it again, and require the round trip to cost no more than 5 word-error-rate points on 20 FLEURS hi_in clips.
The first run failed
Run 1, on a Nebius Serverless AI Job (CPU, 4vcpu-16gb): 8.74 % → 14.68 % WER, +5.94 points. The gate said +5. I missed it by 0.94.
Averages hide the cause, so I looked at the clips. Two of the twenty were very quiet (their loudest frames sat around −46 and −47 dBFS). After the round trip their word error went from 8.3 % to 62.5 % and from 38.9 % to 72.2 %: whole sentences gone. My guess for the first-word slips is that the causal decoder starts cold; lead frames alone did not help (+4.01 vs +2.61 points on the dev set). Thirteen of the twenty clips came back with no change in word error, five others added 6–17 points, and the two quiet ones added 33 and 54.
The fix is one line of audio engineering
My guess is that the codec expects speech at a normal level (its card does not say). So I measure the speech-active level of the input, bring it to −20 dBFS before encoding, store the gain in the file header, and divide it out on decode:
def codec_gain_db(x22, mode: str = "always") -> float:
hop = 551 # 25 ms at 22.05 kHz
frames = x22[: len(x22) // hop * hop].astype(np.float64).reshape(-1, hop)
power = (frames ** 2).mean(axis=1) + 1e-12
db = 10 * np.log10(power)
active = power[db >= db.max() - 30] # speech frames only
rms_db = 10 * np.log10(active.mean())
gain = min(MAX_GAIN_DB, max(MIN_GAIN_DB, TARGET_DBFS - rms_db))
peak = float(np.abs(x22).max()) or 1e-9
return round(float(min(gain, 20 * np.log10(PEAK / peak))), 2) # never clip
I did not choose the setting on the benchmark clips. I ran an ablation on a disjoint dev set of 20 clips, with the rule fixed before the run (lowest round-trip WER delta): always-normalise plus three lead frames won with +1.60 points; the "only if quiet" variant was +2.20, and the run-1 pipeline (no gain, no lead frames) +2.61 on that set.
Run 2 on the 20 benchmark clips: 8.74 % → 12.83 %, +4.09 points. The gate passed. On a second held-out set (fleurs20b), never used for any decision: 12.42 % → 14.46 %.
It is a better average, not a fix for every recording. One normal-level clip got worse (13.0 % → 47.8 %), and I left that in the README. The first run's JSON is committed beside the passing one.
What the numbers buy
Stored size: 815,423 bytes an hour of voice (13 codebooks × 12.5 frames a second, packed 11 bits each), 195× smaller than 22.05 kHz 16-bit PCM. ASR real-time factor 0.16 on 4 vCPU, peak memory 6.9 GB, $0.1345 per audio-hour of compute (the whole-pipeline real-time factor 1.02 × the cpu-d3 list price, with start-up and model load excluded, which is most of the wall time: a note takes about 6.6 minutes from upload to kept at the median, cold start included).
The gate is not only a benchmark. In the product, the worker re-transcribes every note's reconstruction, and the app deletes the raw upload only if word drift is ≤ 5 % (and no 8-word-or-longer segment is over 30 %). Then it asks the bucket for the original and shows the HEAD … 404. The 60-second Opus note I use for the live runs drifts 14.7 %, so its original is kept. That is the gate working.
Two more walls, briefly
NanoCodec's checkpoint config also builds training-only parts: discriminators (one downloads microsoft/wavlm-base-plus) and a speaker encoder. Overriding the config drops them without changing a weight, and a fresh load goes from 45–50 s to 26 s with bit-identical codes:
cfg.discriminator = None
cfg.use_scl_loss = False
cfg.use_slm_loss = False
codec = codec_cls.restore_from(restore_path=path, override_config_path=cfg,
map_location="cpu", strict=False).eval()
And the Jobs API's 64 KiB limit applies to all injected files together (base64), not per file as the docs say. Three files that were each far under the limit were refused at 65,804 bytes of base64 in total. I gzip the worker and run.sh unpacks it. There is more of this in the repo's FEEDBACK.md, each item with a Job or request ID.
The model proposes, code decides
Reminders come from Nemotron 3 Super on Nebius Token Factory with a strict json_schema. Its card does not list Hindi, so nothing it says is shown unless code agrees: the quote must be a verbatim substring of the transcript, the amount is parsed from the quote's own number words, and the date is resolved from the quote and the note's own timestamp. In the live check, one proposal ("Thursday") whose quote never said Thursday had its date removed by code.
export function quoteGate(quote: string, segment: number, segments: Segment[]): QuoteGate {
const q = normalizeHi(quote);
if (q.split(" ").filter(Boolean).length < MIN_QUOTE_TOKENS) return { ok: false, reason: "quote_too_short" };
// try the claimed segment, then its neighbours, then every segment
...
return { ok: false, reason: "quote_not_found" };
}
Honest limits
CPU speed is ours to measure; neither NVIDIA card states CPU support. Time to kept is minutes. The round trip is lossy. Reminders are Hindi-only. The calendar feed is untested on a real phone, and the judge sample is FLEURS read speech, not a family's voice: no volunteer has handed me real notes yet.
Try it without an account: https://hearth.edycu.dev/judge · proof (Job IDs, the raw-audio 404): https://hearth.edycu.dev/proof · code (MIT): https://github.com/edycutjong/hearth · Devpost entry: https://devpost.com/software/hearth-4n0pw5
Top comments (0)