đź“ť Originally published (in Japanese) at forge.workstyle.tech.
I used to think that 80 GB of VRAM would be enough to chat with a large model. In reality, I could only effectively run a 3B-parameter model—and that was without voice.
The full model is 35B parameters. However, only a 3B worth of weights are actively used for each generated token. It took me quite a while to realize that I had been treating "will it fit in memory?" and "is it usable for conversation?" as the same problem.
I’m using an H100 GPU with 80 GB of VRAM. It costs roughly 5 million JPY (~$35,000 USD at current rates). I assumed that with this much hardware, I could comfortably run a large model and hold a conversation. At the very least, I thought I wouldn’t run out of space to load the model.
But being able to run the model is only part of what makes a conversation work. For voice assistants and AI avatars, the fluency of the dialogue depends on response latency. If there’s a long delay between asking a question and receiving an answer, even an intelligent reply can make the conversation feel unnatural. Large models require significant computation per token, which can lead to slower responses.
This is when I realized I had been judging model size solely based on memory capacity. Whether a model is usable for conversation isn’t determined just by whether it fits in memory. The amount of computation required to generate each response matters just as much.
Does Making It Smaller Make Conversations Lighter?
I wanted to speed up responses, so I tried smaller models. If computational costs could be reduced, they might also be suitable for conversational responses. That was my hope.
However, what I was building was a conversation routing classifier that determines the "type of request" based on a user's single utterance. It needs to capture the meaning of the words and distinguish the type of request. Even a short utterance based on the previous action must be interpreted as a single request.
With smaller models, misunderstandings of words became noticeable. For example, "This time, run the same thing with --watch attached". This utterance depends on what was done just before. Looking only at the text and processing it as a standalone new request makes it easy to miss the intent. In my classification, the difference between models was particularly pronounced in such "continuations of the previous action."
Simply making it smaller couldn't maintain both the desired speed of response and the ability to read intent. Then, is there a way to keep the overall model large but reduce computation per word? That's when "MoE (Mixture of Experts)," which is currently attracting attention, became a candidate.
35B Model Uses Only 3B per Word
MoE is a mechanism that enables LLMs to reason efficiently by selecting the optimal expert network from multiple expert networks based on the input. It has multiple specialized sub-networks (experts) and selectively activates only the necessary ones in response to the input.
We adopted Qwen3.6-35B-A3B. Although the entire model has 35B weights, it only uses 3B weights when processing each word.
In ordinary large models, many calculations are required for each word's processing. MoE, on the other hand, keeps the entire model in memory while using only the weights of the selected expert for each word's calculation. The computational cost of this model is said to be similar to that of ordinary 4B and 8B models. In my environment, I am running it on my own gateway (vLLM).
Looking at the numbers, the reason for this choice becomes clear. In the judgment to distinguish what kind of request the user's utterance is, Qwen3.6-35B-A3B had an accuracy rate of 92.6%. Ordinary qwen3:8b and qwen3.5:4b had accuracy rates in the 82% range. The judgment time for 35B-A3B was 306ms.
In particular, the difference was significant for utterances that were a continuation of the previous operation. 35B-A3B had an accuracy rate of 86.3%, while smaller 4B and 8B models had an accuracy rate of 70.4%. In conversations, shorter words often rely on the previous exchange. In the range I observed, not mistaking such utterances became a reason to choose the model.
However, these numbers only show the results of character-based judgment and do not include speech recognition (STT) or speech synthesis (TTS). If we were to have a conversation using voice, the time and GPU memory for that would be added on top. It is difficult to quantify how much it would increase. Based on these judgment results alone, it is also impossible to explain the overall speed of voice conversations.
Fitting in Memory vs. Being Us in Conversation
Previously, I wrote an article titled "You can fit up to four AI avatars on a single GPU." That was regarding maintaining 720p30 in real-time. However, even back then, the GPU was not the bottleneck.
When talking about LLMs, if you only look at GPU memory, you'll miss something. Can you fit the model into 80GB? How much computation is required per token? What is the latency between receiving text and sending a response back? Furthermore, if you are communicating via voice, you must also add the time and memory overhead of STT (Speech-to-Text) and TTS (Text-to-Speech). Even when using the same GPU, many factors determine whether the system is actually viable for a conversation.
What I learned from this configuration is that GPU capacity and conversational responsiveness must be considered from different perspectives. Even with a 35B total model, if the computation per token was equivalent to a 3B model, it remained a viable candidate. Simply switching to a smaller model wasn't enough to maintain the accuracy of the results.
When choosing a GPU, focus on "how much computation is done per token" rather than just the size of the memory.
Top comments (0)