When you pick a speech-to-text model for a product, the first question is not which one hears better. It is whether your transcripts should keep every "uh, you know" the speaker said. We ran into that while putting Underdog Bark 3.0 Pro next to Qwen3-ASR-1.7B on one RTX 3090 on our cloud.
On 40 English clips drawn from the Open ASR Leaderboard test data, Bark got 6.28% of the words wrong and Qwen3-ASR-1.7B got 7.03%. Bark is 1,254 MB on disk, Qwen3-ASR-1.7B is 4,699 MB. Bark is the third link in a chain: NVIDIA's Parakeet TDT 0.6B v3 is the base, Moondream improved it, and Conway Research post-trained and compressed the result. The clips and the leaderboard itself come from the hf-audio team. We packaged it and did the counting. The per-set numbers and the method are in the full Bark 3.0 Pro vs Qwen3-ASR test write-up. Here is what matters when you build on it.
Split the error rate before you trust it
Word error rate adds up words heard wrong, words left out and words added. The total hides which one you have:
| Heard wrong | Left out | Added | Total | |
|---|---|---|---|---|
| Bark 3.0 Pro | 32 | 11 | 15 | 6.28% |
| Qwen3-ASR-1.7B | 28 | 29 | 8 | 7.03% |
Qwen3-ASR actually heard slightly better. It lost on words left out, and most of those were fillers and repeats: "you know", "on, on", "how, how". The leaderboard's human answer keys keep those words. Bark is verbatim on purpose. Its authors write that it "does not clean speech up". Qwen3-ASR tends to tidy them away.
Before comparing, we ran both texts through the English normalizer OpenAI wrote for Whisper, a cleanup pass that drops case, punctuation and "uh" or "um", then scored with jiwer. If your product wants clean text, build references in that style, or the benchmark rewards the wrong habit. Being literal cuts both ways: Bark writes stutters such as "h-how" as heard, and the scorer marks them wrong.
Only 924 words were scored, so a couple of words can move a set. Call it a near tie that leans toward Bark.
Why it needs its own backend
The usual serving programs, vLLM, SGLang and llama.cpp, load a model and answer requests, but none runs Parakeet TDT models. So Bark gets a custom backend: one Python file, under 200 lines, built on Hugging Face's Transformers library, as the model page says. Any client of OpenAI's transcription API can call it. The weights load in bf16, the 16-bit number format they are published in. Each recording gets 0.25 s of silence in front, as the model card recommends. Decoding is greedy: the model takes its single most likely output at each step instead of weighing alternatives.
The trap was dependencies. Bark needs Transformers 5.19.0, and the qwen-asr package pins 4.57.6. With no shared environment, Qwen3-ASR ran on SGLang, so any speed comparison compares setups as much as models.
The setting that decides your GPU is chunk length
Peak memory allocated by PyTorch, the library underneath Transformers, was 1.243 GiB on the short clips. The five-minute file, sent as one request, went to 2.151 GiB.
We capped the 3090 at about 2 GB to imitate a smaller card. The clips passed. The five-minute file failed with CUDA out of memory. Tried to allocate 476.00 MiB ... 2.00 GiB allowed. Cutting long audio into 60-second pieces fixed it: peak 1.369 GiB, the same transcripts on all 40 clips, and no slowdown. Our example defaults to 600-second pieces, so lower it for small cards.
The same knobs run it with no GPU at all:
BARK_DEVICE=cpu
BARK_CPU_THREADS=4
BARK_CHUNK_SECONDS=60
With four CPU threads on a server, the five-minute file finished in 31 seconds; sixteen threads cut that to 14. Output was identical to the GPU run on every one of the 40 clips. Changing any of these variables needs a delete and a fresh deployment.
A cap mimics how little memory a small card has, but not how slow it is, and none of the seven small card types we tried to rent was in stock. So we cannot name one as enough.
What to measure on your own audio
Measure these:
- Errors split into heard wrong, left out and added, against references in your own style.
- GPU time and round trip (until the answer is back) separately. Bark needed about 0.12 seconds of GPU work per clip, while a typical clip took half a second round trip. The network was the slow part.
- Peak memory on your longest real file.
- Whether the whole transcript comes back. On the five-minute file, Qwen3-ASR on SGLang with default settings gave back 212 words and then quit. We have not dug into the cause.
On InferenceKey, transcription uploads stop at about 2 MB, roughly 65 seconds of uncompressed audio, so a 9.7 MB WAV was rejected. The same five minutes as compressed audio fit in 1.1 MB. That limit is open, and the SDK example and its README in the Bark research post document the workaround.
What it does not tell you
We did not reproduce the model card's leaderboard score of 4.29%, which is self-reported. We used the original, uncleaned test sets, so our absolute numbers do not compare with the card's table, only the two models with each other. One real GPU. Bark is English only, and its authors say it has no word timings or speaker names yet. If you need those, our test of Nemotron 3.5 Transcribe with timed, speaker-tagged output covers a model that does.
The full measurements and the step-by-step setup are in the Underdog Bark 3.0 Pro speech-to-text test on inferencekey.com. Thanks to Conway Research, the Underdog team, Moondream, NVIDIA, the Qwen team and the hf-audio team for the models and the data, and to the AI Search video where we spotted Bark.
Top comments (0)