DEV Community

#llamacpp

Posts

đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.
Hands-On: Qwen3.8-Flash-Next and a Glimpse into Qwen4's Design Philosophy — llama.cpp Support Status and A100 Benchmarks
oooocean66 profile

Hands-On: Qwen3.8-Flash-Next and a Glimpse into Qwen4's Design Philosophy — llama.cpp Support Status and A100 Benchmarks

#ai #llm #llamacpp #qwen
Comments
20 min read
Fine-Tuning Gemma 4 on an 8 GB RTX 3050: Slow, Low End and Still Viable

Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady Submission 🦥
xbill profile

Fine-Tuning Gemma 4 on an 8 GB RTX 3050: Slow, Low End and Still Viable

#devchallenge #hf26challenge #gemma #llamacpp
4
Comments
14 min read
Extending llama.cpp to Support Reflection Beam-1.0-501B-A23B: Handling Complex MoE and Attention Architectures
pdrojack profile

Extending llama.cpp to Support Reflection Beam-1.0-501B-A23B: Handling Complex MoE and Attention Architectures

#llamacpp #mixtureofexperts #attention #modelconversion
Comments
3 min read
Qwen and Gemma With Thinking On: Thinking Pays Qwen on Tool Calls, Gemma Still Leads
xbill profile

Qwen and Gemma With Thinking On: Thinking Pays Qwen on Tool Calls, Gemma Still Leads

#gemma #qwen #llamacpp #cuda
1
Comments 1
9 min read
How Much Context Fits Beside a 12B on an 8 GB GPU? Gemma 4 12B's Context Cache on an RTX 3050
xbill profile

How Much Context Fits Beside a 12B on an 8 GB GPU? Gemma 4 12B's Context Cache on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
2
Comments
9 min read
Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp
eme_gug_0821b41b948be6516 profile

Running MoE LLMs Larger Than Your RAM: A Practical Guide to mmap, Expert Offloading and SSD Streaming with llama.cpp

#llm #llamacpp #moe #performance
Comments
8 min read
Do Drafters Work on Small Models? Gemma 4 E2B and E4B on Two Small GPUs

Acceptance rates hold steady across hardware
xbill profile

Do Drafters Work on Small Models? Gemma 4 E2B and E4B on Two Small GPUs

#gemma #llamacpp #cuda #benchmarking
8
Comments 3
19 min read
A Dense 31B on an 8 GB GPU: Gemma 4 31B on an RTX 3050
xbill profile

A Dense 31B on an 8 GB GPU: Gemma 4 31B on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
10
Comments
10 min read
llama-server's prompt cache reuses KV computed under a different LoRA scale
homelabpm profile

llama-server's prompt cache reuses KV computed under a different LoRA scale

#llamacpp #llm #selfhosted #lora
Comments
6 min read
Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)
kharlampiiev profile

Fine-tuning Qwen2.5-1.5B on a Mac to fix a parroting chatbot (and halve the prompt)

#ai #llm #machinelearning #llamacpp
Comments
6 min read
Comparing Qwen and Gemma on the Low End: Qwen Is Faster Without Drafting, Gemma Has Better Quality
xbill profile

Comparing Qwen and Gemma on the Low End: Qwen Is Faster Without Drafting, Gemma Has Better Quality

#gemma #qwen #llamacpp #cuda
14
Comments
13 min read
Can a QAT 12B Fit an 8 GB GPU? Gemma 4 12B on an RTX 3050
xbill profile

Can a QAT 12B Fit an 8 GB GPU? Gemma 4 12B on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
7
Comments
15 min read
Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE Models on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #machinelearning
Comments
8 min read
Can a 26B MoE Run on an 8 GB GPU? Gemma 4 26B A4B on an RTX 3050

RAM bottlenecks and speeds inside
xbill profile

Can a 26B MoE Run on an 8 GB GPU? Gemma 4 26B A4B on an RTX 3050

#gemma #llamacpp #cuda #benchmarking
12
Comments 10
11 min read
Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp
eme_gug_0821b41b948be6516 profile

Running 100B+ MoE LLMs on a Single RTX 4090: A Practical Guide to Expert Offloading with llama.cpp

#llm #llamacpp #gpu #selfhosted
Comments
7 min read
đź‘‹ Sign in for the ability to sort posts by relevant, latest, or top.