Speech pipelines usually chain separate tools: text to speech, cloning, noise removal, speaker separation. AuK-Flash replaces the chain with one model driven by a sentence. You send Replace 'twelve percent' with 'eighteen percent'. and a recording, and the same voice comes back saying eighteen. On an NVIDIA RTX A5000 that edit kept the graphics card (GPU) busy for 0.73 seconds.
AuK and AuK-Flash come from Ziyang Ma and 32 co-authors at Shanghai Jiao Tong University, Tencent Hunyuan and the Shanghai Innovation Institute. Our AuK-Flash test on a 24 GB card has the measurements. This piece covers the decisions you face wrapping it as an API.
Size the card for two models, not one
The AuK-Flash page lists about 6.8 GB. That is not what you deploy. AuK understands your instruction through a second model, Alibaba's Qwen2.5-Omni-3B (the "encoder"), which weighs about 12 GB more. A working install pulls 18.75 GB.
The memory on the card (VRAM) follows. We saw about 15.6 GB in use once loaded and about 20.3 GB at peak. Our own example's .env.example says "about 17 GB of free VRAM", which invites a 16 GB card. Plan for 24 GB.
A CPU offload setting, which parks part of the model in ordinary system RAM, cut the peak to about 11.3 GB. Each request then took 13 to 15 seconds, longer than the audio it produced. It might fit a 16 GB card, untested.
The encoder also carries the licence question. AuK's code and weights are MIT, but Qwen2.5-Omni-3B is under the Qwen Research Licence, which does not allow commercial use. On issue #13, project collaborator ZhikangNiu replied that the team plans a permissively licensed encoder in a future release. Settle this before a product depends on it.
The sampling knobs are locked, on purpose
AuK sharpens audio from a blurry sketch in repeated passes. The base model does 32, each also computing an unguided version to push the result toward your instruction (that push is "guidance"). AuK-Flash was trained to need 4 passes and no extra version: 4.5 times faster, by the authors' measurement.
The inference code (what runs the trained model on each request) hard-codes that recipe and ignores any step count or guidance strength you pass Flash. A source comment says re-enabling guidance "blows up the amplitude". If your wrapper exposes those parameters, hide them or label them base-model only.
Prompt wording is part of your API contract
In voice design you describe a voice in text and give it a line. Two phrasings that look interchangeable behave very differently:
Based on the following description: "...", generate speech content "...".
Generate speech based on the following description: "...". The content to speak is: "...".
The first comes from the project's command-line example: 7 of 8 English voice designs came out exactly right. The second is the cookbook's English template, and in 8 of 8 tries the model spoke part of the description out loud. The technical report recommends a separate language model in front to rewrite requests into the trained phrasing. We ran without one.
So never forward free-form text. Keep fixed templates on the server, fill the slots, and test each template like an endpoint.
One model is one queue
A single model is one artifact to ship and patch, and adding a task means writing an instruction, not integrating a tool. The price is concurrency (requests running at once). Our adapter, the code that loads the model and answers requests, allows one GPU job at a time: four edits fired together came back in turn, the last after 5.8 seconds. Warm calls (model loaded, first request done) used 0.4 to 0.8 seconds of GPU, so short edits queue fine. A long job blocks the next caller.
The cold start, the time from asking for the model to it being ready to serve, was 71.6 seconds on a GPU machine already connected to InferenceKey with no model files on it. The cold start and per-task timings for AuK-Flash break that down step by step.
Measure the words, then admit what a transcript can't see
We transcribed every output with OpenAI's Whisper (large-v3-turbo) and diffed the text. No edit touched the words around the change: a cheap regression test for any speech editing service.
Text cannot tell you whether the pitch went up, the whisper sounds like one, or the swapped word has an audible join. That needs an ear or a pitch measurement, and we used neither. Our inputs were synthetic voices from Kyutai's Pocket TTS (see our Pocket TTS Spanish voice on CPU), the friendliest possible case. Noisy rooms and overlapping speakers remain untested. Training covered Chinese and English only; our Spanish attempt was partly garbled. Benchmark scores (87.5 correct edits in 100 on SpeechEditBench) are the authors', not reproduced. Our run was around 60 requests, nearly all single attempts, none crashed.
What our part was
No ready-made InferenceKey backend handles AuK yet, so we wrote that small adapter ourselves. It pins Tencent's code, caps outputs at 30 seconds and answers on the chat address, because our text-to-speech address cannot yet return audio from a custom adapter, a gap on our side.
The AuK team built both models and wrote the report and the cookbook in the AuK repository. SGLang-Omni, vLLM-Omni and 0xShug0's audio.cpp each added ways to run AuK. Alibaba's Qwen team made the encoder. Issue reporters raised the licence, Polish and shouting questions. We spotted the release in AI Search's news video.
The full AuK-Flash write-up on inferencekey.com has the per-task table, every audio sample before and after, the platform issues we hit and the SDK example that chains a design, clone, edit and speed-up.
Top comments (0)