Computer science problems and solutions, tips on free and open source software (FOSS), programming, artificial intelligence, self-hosting, Meshtastic/LoRA, hardware, science, and tech in general
Try the qwen2.5-coder model family. Yes, 4GB is insufficient to run anything useful. I'm trying with qwen2.5-coder:14b-instruct-q2_K (so low quantization and higher parameters) and it's not bad at all. The speed and quality is decent all considering. You'll need about 20GB of RAM, however. Be aware I got Chinese language only replies when running 1.5B models of that family.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
Computer science problems and solutions, tips on free and open source software (FOSS), programming, artificial intelligence, self-hosting, Meshtastic/LoRA, hardware, science, and tech in general
ollama.com/library/qwen2.5-coder/tags You'll have to experiment the smallest models with different quantization levels and avoid swapping to disk during inference.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
AI Software Engineer 🚀 | MSc Computer Science | Building scalable cloud & AI systems | Open-source enthusiast | Writing about code, career & the chaos in between
Thanks for being clear about the hardware limits. Many people try to run local LLMs, thinking it will just work, then get frustrated when it is slow or crashes. Posts like this help save a lot of time and confusion.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
I’m a web developer who loves turning ideas into stunning, interactive websites. I’m all about creating websites that not only look great but also work seamlessly.
Try the
qwen2.5-codermodel family. Yes, 4GB is insufficient to run anything useful. I'm trying withqwen2.5-coder:14b-instruct-q2_K(so low quantization and higher parameters) and it's not bad at all. The speed and quality is decent all considering. You'll need about 20GB of RAM, however. Be aware I got Chinese language only replies when running 1.5B models of that family.Thanks for the tip! I’ll definitely check out
qwen2.5-coderollama.com/library/qwen2.5-coder/tags You'll have to experiment the smallest models with different quantization levels and avoid swapping to disk during inference.
Thanks for the link! I’ll explore the smallest models and test their performance. Thank you for the suggestion!
Thanks for being clear about the hardware limits. Many people try to run local LLMs, thinking it will just work, then get frustrated when it is slow or crashes. Posts like this help save a lot of time and confusion.
Appreciate that! I'm glad the post was helpful.
Totally get the RAM struggle with local LLMs, I had a similar bottleneck running anything larger than a 3B model too.
Have you found any tricks to make chat-style workflows smoother in the CLI, or do you just keep it basic?
Since I'm still learning the concept and getting an understanding of how everything works, I'm sticking to the basics for now
Ollama is Great. You can also use Docker Model Runner for this
Yeah, ollama is a valuable tool. Thanks for sharing.
This is extremely impressive, love how you documented the process and called out the RAM struggle directly. Makes me wanna try it on my old laptop now
Thank you for the appreciation. Ollama is definitely worth a try.
Yes, I will check out the smaller models. Thanks for the useful advice.