DEV Community

Cover image for Phi-3-mini on Linux ARM: Run LLMs Locally with MLC LLM
KAMAL KISHOR
KAMAL KISHOR

Posted on

Phi-3-mini on Linux ARM: Run LLMs Locally with MLC LLM

#ai #linux #llm #machinelearning

Running large language models often feels like a cloud-only affair, requiring powerful GPUs and substantial budgets. But what if you want to run a capable LLM like Microsoft's Phi-3-mini on a low-power Linux ARM device, say, a Raspberry Pi 5 or an ARM laptop, without relying on cloud APIs or a beefy discrete GPU? This is where projects like MLC LLM come into play, offering a path to efficient local inference.

What You'll Be Running and What You Need
We're going to set up and run Phi-3-mini-4k-instruct, a compact yet powerful model from Microsoft, directly on your ARM-based Linux machine. Phi-3-mini is notable for its small size (3.8 billion parameters) and strong performance for its scale, making it ideal for edge deployments. To make this happen, we'll use MLC LLM (Machine Learning Compilation for Large Language Models), an open-source universal deployment framework that compiles LLMs to run efficiently on various hardware, including ARM CPUs.

You'll need:

  • An ARM-based Linux device (e.g., Raspberry Pi 5, Pinebook, or an ARM-based laptop).
  • A recent version of Python (3.10+ recommended).
  • pip for package management.
  • Sufficient storage for the model (~6GB for Phi-3-mini-4k-instruct).

Setting Up MLC LLM

First, ensure you have Python and pip ready. For most Linux distributions, you can install these via your package manager. Once that's covered, installing MLC LLM is relatively straightforward using pip. MLC LLM provides pre-built wheels for common architectures, which simplifies the process.

Let's install the mlc_llm and mlc_chat packages:

pip install --pre -U -f https://mlc.ai/wheels mlc-llm-nightly mlc-chat-nightly
Enter fullscreen mode Exit fullscreen mode

This command installs the nightly builds, which often include the latest model support and optimizations.
After installation, the mlc_chat command-line interface becomes available, allowing you to download models and start conversations.

Downloading and Running Phi-3-mini

With MLC LLM installed, you can now download the Phi-3-mini model. MLC LLM manages models in a structured way, often downloading them into a local mlc-chat-config directory.

To start the chat and automatically download Phi-3-mini-4k-instruct (using the q4f16_1 quantization, which offers a good balance of size and performance), use the following command:

mlc_chat chat Phi-3-mini-4k-instruct-q4f16_1
Enter fullscreen mode Exit fullscreen mode

This command will first check for the model. If it's not present, it will download the necessary model weights and compiled library for your ARM architecture. This might take some time depending on your internet connection. Once downloaded, the chat interface will launch in your terminal.

A Quick Example

After the model loads, you'll see a prompt similar to a regular chat interface. You can type your query, and Phi-3-mini will respond.

[system] user: Tell me a short story about a brave knight and a dragon.
Enter fullscreen mode Exit fullscreen mode

Phi-3-mini will then generate a response:

[assistant] In a land of ancient forests and towering peaks, Sir Kaelin, known not for his strength but his unwavering heart, faced the formidable task of confronting Ignis, the fire-breathing dragon. The beast had slumbered for decades, but now its shadow fell upon the kingdom. Kaelin, armed with only a shield bearing his family's crest and a rusty sword, ventured into the dragon's lair. He found Ignis not in a rage, but guarding a single, unhatched egg. Kaelin, understanding the dragon's sorrow, chose not to fight, but to speak. He promised the kingdom would protect the egg if Ignis would retreat to the highest mountain. A silent understanding passed between them. Ignis, with a final, mournful glance at the egg, took to the sky, and Kaelin returned, not with a dragon's head, but with peace.
Enter fullscreen mode Exit fullscreen mode

The exact output will vary, of course, but you'll experience Phi-3-mini generating text directly on your ARM device.

Common Problems and Considerations

  • Slow Download Speeds: Model downloads can be large (several gigabytes). A stable and fast internet connection is helpful.
  • Insufficient Memory: While Phi-3-mini is small, running LLMs still requires a decent amount of RAM. Ensure your ARM device has at least 4GB of RAM; 8GB is better for a smoother experience. The Raspberry Pi 5 with 8GB RAM is a good candidate.
  • Unsupported Architecture: MLC LLM aims for broad support, but ensure your specific ARM processor (CPU architecture and instruction sets) is compatible with the pre-built binaries. If you encounter errors, building MLC LLM from source might be an option, though it's a more involved process.
  • Performance Expectations: Running a model locally on a CPU, especially an ARM CPU without a dedicated GPU, will not be as fast as a powerful GPU-accelerated setup. Expect a noticeable delay in response generation. However, for many local, privacy-focused applications, this trade-off is acceptable. As someone who prefers free, local AI tools when the quality is good enough (I even wrote about replacing Cursor with a free local coding assistant), I find this performance level often perfectly suitable for offline tasks or light agentic workflows. For similar local LLM setups on mobile-adjacent hardware, you might also find my guide on Running Llama 3.2 on Android: A Step-by-Step Guide Using Ollama useful.
  • Model Availability: The mlc_chat command relies on models and pre-compiled libraries being available from the MLC AI community. Occasionally, a specific model or quantization might be in flux, leading to download issues. Check the MLC LLM GitHub issues for updates.

One important limitation here is that while MLC LLM enables CPU inference on ARM, for real-time applications or heavy batch processing, a dedicated GPU is still the gold standard. If your use case demands sub-second response times for complex queries or needs to process many requests concurrently, even a small LLM like Phi-3-mini will likely struggle on a low-power ARM CPU alone. For such scenarios, investing in hardware with GPU acceleration or utilizing cloud-based services becomes more practical.

Sources

  1. https://github.com/mlc-ai/mlc-llm
  2. https://huggingface.co/microsoft/Phi-3-mini-4k-instruct
  3. https://github.com/mlc-ai/mlc-llm#getting-started
  4. https://github.com/mlc-ai/mlc-llm/blob/main/docs/get_started/mlc_chat_cli.md

This article was generated with AI (Google Gemini + web search). Please check important details against the sources above.

Daily AI notes on LinkedIn — Kamal Kishor

Top comments (0)