DEV Community

#cuda

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp

Includes a tricky broadcast K/V debugging story
xbill profile

Gemma 4 on a 4 GB GPU in Plain PyTorch: From Out of Memory to Faster Than llama.cpp

#gemma #pytorch #cuda #quantization
2
Comments 3
13 min read
Project EREN: My Journey to Building a CUDA-Powered LLM Inference Engine
pathakarunav123 profile

Project EREN: My Journey to Building a CUDA-Powered LLM Inference Engine

#ai #llm #cuda #opensource
1
Comments
4 min read
TorchTPU vs CUDA: Can Native PyTorch Make Google TPUs Easier to Use?
jamilxt profile

TorchTPU vs CUDA: Can Native PyTorch Make Google TPUs Easier to Use?

#pytorch #tpu #cuda #ai
1
Comments
8 min read
Amazon: New TorchNeuron tool challenges Nvidia's dominance
hacksgr profile

Amazon: New TorchNeuron tool challenges Nvidia's dominance

#torchneuron #pytorch #trainium #cuda
1
Comments
2 min read
Running NVIDIA's new 120B Nemotron-3.5-Super on an 8 GB GPU — skipping the zeros, and finding the real bottleneck
helgard_orlm profile

Running NVIDIA's new 120B Nemotron-3.5-Super on an 8 GB GPU — skipping the zeros, and finding the real bottleneck

#ai #llm #cuda #python
Comments
7 min read
Qwen and Gemma With Thinking On: Thinking Pays Qwen on Tool Calls, Gemma Still Leads
xbill profile

Qwen and Gemma With Thinking On: Thinking Pays Qwen on Tool Calls, Gemma Still Leads

#gemma #qwen #llamacpp #cuda
1
Comments 1
9 min read
How Much Context Fits Beside a 12B on an 8 GB GPU? Gemma 4 12B's Context Cache on an RTX 3050
xbill profile

How Much Context Fits Beside a 12B on an 8 GB GPU? Gemma 4 12B's Context Cache on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
2
Comments
9 min read
I wrote 21 CUDA kernels on a free GPU. Here's what the profiler taught me.
musharaf_khanpathan_90de profile

I wrote 21 CUDA kernels on a free GPU. Here's what the profiler taught me.

#cuda #gpu #performance #cpp
Comments
5 min read
Running vLLM natively on Windows
aivrar profile
aivrar

Running vLLM natively on Windows

#vllm #windows #cuda #llm
Comments
6 min read
A Dense 31B on an 8 GB GPU: Gemma 4 31B on an RTX 3050
xbill profile

A Dense 31B on an 8 GB GPU: Gemma 4 31B on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
10
Comments
10 min read
Can a QAT 12B Fit an 8 GB GPU? Gemma 4 12B on an RTX 3050
xbill profile

Can a QAT 12B Fit an 8 GB GPU? Gemma 4 12B on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
7
Comments
15 min read
Running the 510 GB DeepSeek-V4.1-Flash on an 8 GB GPU — and three bugs that never raise an error
helgard_orlm profile

Running the 510 GB DeepSeek-V4.1-Flash on an 8 GB GPU — and three bugs that never raise an error

#ai #llm #cuda #python
Comments
8 min read
Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe
helgard_orlm profile

Running a 133 GB MoE model on an 8 GB GPU at 11 tokens/s by streaming experts from NVMe

#ai #llm #cuda #python
Comments
6 min read
How Fast Can a 421M-Parameter Decision Model Run? I Benchmarked Laya Across NVIDIA GPUs
cookies_c9dc8b91f33d29250 profile

How Fast Can a 421M-Parameter Decision Model Run? I Benchmarked Laya Across NVIDIA GPUs

#machinelearning #cuda #performance #opensource
Comments 1
6 min read
Can a 26B MoE Run on an 8 GB GPU? Gemma 4 26B A4B on an RTX 3050

RAM bottlenecks and speeds inside
xbill profile

Can a 26B MoE Run on an 8 GB GPU? Gemma 4 26B A4B on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
12
Comments 11
11 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.