nixamp remembers what it heard, and can say it in Swedish

Earlier this week nixamp learned to hear. Now it remembers what it heard, and can say it in Swedish.

Written down once

When a channel is captioned, the server carrying it sends five seconds of sound at a time to nixamp.com's ear, an open Whisper model running on our own CPU. That worked, but it had no memory. Play the same film tomorrow and the ear heard the whole thing again. Join the same live from two servers and both paid for every word.

nixamp 0.24.0 keeps what it hears. The transcript is stored under the identity of the media, not the channel that happened to play it: a file by a fingerprint of its bytes, a link by its address, a live as the one broadcast it was. Lines are seconds into the media. The next captioner to meet the same film asks the store first and reads any moment it already knows instead of waking the ear. What it hears beyond that gets added. Over time the store fills in and the ear goes quiet.

nixamp transcribe FILE now does the whole thing in one pass, a minute at a time, with the timing of each line, and marks the transcript complete. Then --srt, --vtt or --out DIR gives you subtitle files. Do it once on any machine and every server that puts that film on the air reads the lines back.

nixamp transcribe film.mkv --translate de,sv --out subs/
nixamp transcript --kept film.mkv --srt
nixamp transcript --list

It knows what language it is hearing

Transformers.js never detects Whisper's language. It quietly assumes English, and a Swedish clip came back as three English words repeated for half a minute. The ear now does what whisper.cpp does: one pass over the first thirty seconds, one decoder step, and the loudest language token wins. About a second, and then a Swedish channel says something in Swedish.

In another language

Ask for a language and the lines come translated, by the Helsinki-NLP OPUS-MT models on nixamp.com's own CPU. Each pair is a hundred megabytes and takes about a second per sentence. German and Swedish were the two we wanted, and anything with a model from or into English works; a pair with no model of its own goes through English. A translation is made once and kept beside the original, so nobody translates the same film twice either.

We measured the big many-language models first. NLLB-200 wanted two gigabytes of memory and ten to twenty seconds a sentence on our box. The small pairs won.

On the page there is a picker beside the Captions switch. In the terminal, nixamp transcript --channel ID --language sv follows a live in Swedish, and nixamp translate --to de "..." does a line.

An MCP for the site

The tools that nixamp mcp speaks on a pipe are now also at https://nixamp.com/mcp over HTTP. An agent with no nixamp installed sends a nixamp token as the bearer and gets the watch parties, the rooms, and the transcripts: transcript_get, transcripts_list, translate_text, and transcribe_audio for a whole film.

The API

GET  /api/v1/transcripts/ID?format=srt        a kept transcript, by id or by media identity
GET  /api/v1/transcripts/ID?language=de       translated once and kept; 202 with progress while a long one is made
POST /api/v1/transcripts/ID/lines             keep lines
GET  /api/channels/ID/captions?language=sv    a live's captions, translated as they are said
POST /api/v1/translate                        texts in another language

Everything is open models on our own hardware. Nothing goes to a speech vendor, nothing is billed, and the models are baked into the image so a deploy never downloads them again.

nixamp is at https://nixamp.com and the code is at https://github.com/profullstack/nixamp.

DEV Community