DEV Community

Cover image for How I squeezed three models into 6 GB of VRAM
Alexey
Alexey

Posted on Edited on

How I squeezed three models into 6 GB of VRAM

#gpu #machinelearning #llm #python

A few days ago my friend and I took part in a hackathon. The case was a web service for video lectures: upload a recording or a YouTube link, get a transcript, synced subtitles, a summary with a table of contents, and search by meaning, so a student can type "where did he prove the Pythagorean theorem" and land on the right second of the video.

We had three days. We took fifth place.

The project is called Lectio and the code is on GitHub. This post isn't really about the hackathon though. It's about one problem that ate a surprising amount of my time: making all of it run on my own PC.

The hardware

  • Ryzen 5 5600
  • 32 GB of RAM
  • GTX 1660 Super with 6 GB of VRAM

The 1660 Super came out in 2019. It's a Turing card without tensor cores. It's fine for games. For running neural networks it is, let's say, modest.

And the whole thing had to work offline. No cloud APIs, everything local.

Three models, one small card

The processing pipeline needs three models:

  1. Whisper large-v3-turbo (through faster-whisper) turns speech into text with a timestamp for every word.
  2. BGE-M3 turns chunks of the transcript into vectors for semantic search.
  3. Qwen3 4B (through Ollama) writes the summary, the table of contents and the quiz.

Each of them fits into 6 GB on its own. All three together don't.

And 6 GB isn't really 6 GB. My desktop takes 1–1.5 GB before I start anything. Discord takes another half a gigabyte. I found that out when Ollama suddenly started running 8–17% of the model on the CPU, and the reason turned out to be Discord sitting in the tray.

The first attempt

The simplest version is to load every model once, keep it in memory and process lectures. On 6 GB that doesn't fit, and Whisper was the first to run out of memory: Ollama was still holding the LLM after a chat request, and there was no room left.

What I noticed next was more interesting than the crash itself. The models behave very differently when there's not enough memory.

Ollama adapts. If the card is busy, it quietly offloads part of the layers to the CPU and keeps working. Slower, but it works.

Whisper and the embedder don't adapt. They try to allocate memory, fail and take the whole job down with them.

So the polite one was getting everything and the strict ones were crashing. That's backwards. The fix was to take memory back from Ollama before my own models load, not the other way around.

Stage by stage

The worker now processes one lecture at a time, and every stage gets the card to itself:

  1. Ask Ollama to unload its model.
  2. Load Whisper, transcribe, unload Whisper.
  3. Ask Ollama to unload again, in case the chat loaded it in the meantime.
  4. Load BGE-M3, build the vectors, unload BGE-M3.
  5. Now Ollama gets the whole card for the summary.

It's all behind one setting, RELEASE_MODELS_BETWEEN_STAGES=true. On a card with more memory you turn it off and nothing gets unloaded.

Ollama doesn't let go

By default Ollama keeps a model loaded for five minutes after the last request. So if someone asked the lecture chat a question, and a minute later a new lecture went into processing, Whisper would start with the LLM still occupying the card.

Ollama has an endpoint for this: send a request with keep_alive: 0 and it unloads the model right away. It isn't part of the OpenAI-compatible API, so I call the native one:

async def unload(self) -> None:
    native_url = self._base_url.removesuffix("/v1") + "/api/generate"

    try:
        async with httpx.AsyncClient(timeout=10.0) as client:
            await client.post(
                native_url,
                json={"model": self._model, "keep_alive": 0},
            )
    except httpx.HTTPError:
        # Other OpenAI-compatible servers don't have this endpoint.
        return
Enter fullscreen mode Exit fullscreen mode

If the LLM is something other than Ollama, the request just fails quietly and processing continues.

Unloading a model is harder than it sounds

Models are created once and cached:

@cache
def get_transcriber() -> Transcriber:
    return FasterWhisperTranscriber()


def release_transcriber() -> None:
    get_transcriber.cache_clear()
    gc.collect()
    torch.cuda.empty_cache()
Enter fullscreen mode Exit fullscreen mode

Clear the cache, run the garbage collector, tell PyTorch to give the memory back. Looks fine.

It didn't free anything.

The problem was in the code that used the model:

model = get_transcriber()
try:
    transcript = model.transcribe(audio_path)
finally:
    release_transcriber()
Enter fullscreen mode Exit fullscreen mode

At the moment release_transcriber() runs, the local variable model still points at the model. The cache is cleared, but the object is alive, so the garbage collector can't touch it and the memory stays taken.

The fix looks weird, but it works. Never keep the model in a variable:

# No local reference to the model: release_* can only free memory
# once nothing else points at it.
try:
    transcript = await asyncio.to_thread(
        lambda: get_transcriber().transcribe(audio_path, progress),
    )
finally:
    release_transcriber()
Enter fullscreen mode Exit fullscreen mode

The comment in the code exists so I don't "clean it up" later.

fp16, but the right way

BGE-M3 in fp32 is too much for this card, so it runs in half precision. My first version was the obvious one:

model = SentenceTransformer("BAAI/bge-m3", device="cuda").half()
Enter fullscreen mode Exit fullscreen mode

Memory usage dropped less than expected. The model was loaded in fp32 first and converted afterwards, and the fp32 copy stayed in PyTorch's CUDA allocator cache.

It has to be loaded in fp16 from the start:

self._model = SentenceTransformer(
    settings.embedding_model,
    device=device,
    model_kwargs={"dtype": torch.float16} if device != "cpu" else None,
)
Enter fullscreen mode Exit fullscreen mode

The trade-off I didn't expect

The API needs its own copy of BGE-M3 to turn search queries into vectors. Another model on the card.

I moved it to the CPU, and the numbers turned out like this:

  • search got slower by about 40 ms per query, which nobody will notice;
  • Qwen3 now fits on the GPU entirely and generates 68 tokens per second instead of 39.

A 40 ms slower search for an LLM that's almost twice as fast. One line in .env:

SEARCH_DEVICE=cpu
Enter fullscreen mode Exit fullscreen mode

The small stuff

A few more settings made a real difference:

WHISPER_COMPUTE_TYPE=int8_float16
WHISPER_BATCH_SIZE=4
LLM_MODEL=qwen3:4b-instruct
Enter fullscreen mode Exit fullscreen mode

Whisper weights in int8 with fp16 computation, a small batch, and a 4B model instead of 8B. For Ollama I also turned on OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=q8_0, which makes the context cache half as big.

And one thing I didn't have time to dig into. faster-whisper runs on CTranslate2, not PyTorch. But CTranslate2 only found cuBLAS and cuDNN when torch had been imported before it. So the transcriber starts with this:

import torch  # noqa: F401
from faster_whisper import BatchedInferencePipeline, WhisperModel
Enter fullscreen mode Exit fullscreen mode

The import isn't used. Remove it and Whisper stops finding CUDA.

The numbers

The evening before the defense I put a 1 hour 25 minute lecture through the pipeline. From upload to "ready" it took 14 minutes 16 seconds:

Stage Time
Extracting audio (ffmpeg) 7 s
Transcription (Whisper) 5 min 2 s
Indexing (BGE-M3) 39 s
Summary (Qwen3 4B) 8 min 27 s

Whisper itself took 4 minutes 56 seconds for 85 minutes of audio. That's about 17 times faster than real time, on a card with no tensor cores.

Search works after six minutes. Everything after that is the 4B model writing the summary. That's 60% of the total time, and that's what a bigger card or a stronger model would speed up the most.

What I took from this

I expected the hard part to be the models. It wasn't. The models are good, the libraries are good, and each of them works fine as long as it's alone on the card.

The hard part was sharing. Which model gives memory back politely and which one crashes. Who holds a reference you forgot about. Which library caches what, and for how long.

None of it is in the model docs, because each model is documented as if it's the only thing running on your GPU.

With 6 GB it never is.

Top comments (2)

Collapse
 
hitler_homeless_0255061af profile image
Hitler Homeless •

Amazing work brother, clan proud of you

Collapse
 
reidmarlow profile image
Reid Marlow •

The allocator trap that usually bites multi-model pipelines on 6GB cards is mixing runtimes. Faster-whisper runs on CTranslate2, while SentenceTransformer runs on PyTorch. Because both maintain separate C++ memory pools outside Python's garbage collector, PyTorch calling empty_cache has zero visibility into CTranslate2 allocations and vice versa. Even with local references cleared, memory fragmentation between the two libraries eventually causes an unrecoverable CUDA out of memory error. Running Whisper in a separate process that exits immediately after the audio file finishes transcription guarantees the driver reclaims every byte before PyTorch attempts to allocate for embeddings.