DEV Community

KAMAL KISHOR
KAMAL KISHOR

Posted on

227

Running Llama 3.2 on Android: A Step-by-Step Guide Using Ollama

Llama 3.2 was recently introduced at Meta’s Developer Conference, showcasing impressive multimodal capabilities and a version optimized for mobile devices using Qualcomm and MediaTek hardware. This breakthrough allows developers to run powerful AI models like Llama 3.2 on mobile devices, paving the way for more efficient, private, and responsive AI applications.

Meta released four variants of Llama 3.2:

  • Multimodal models with 11 billion (11B) and 90 billion (90B) parameters.
  • Text-only models with 1 billion (1B) and 3 billion (3B) parameters.

The larger models, especially the 11B and 90B variants, excel in tasks like image understanding and chart reasoning, often outperforming other models like Claude 3 Haiku and even competing with GPT-4o-mini in certain cases. On the other hand, the lightweight 1B and 3B models are designed for text generation and multilingual capabilities, making them ideal for on-device applications where privacy and efficiency are key.

In this guide, we'll show you how to run Llama 3.2 on an Android device using Termux and Ollama. Termux provides a Linux environment on Android, and Ollama helps in managing and running large models locally.

Why Run Llama 3.2 Locally?

Running AI models locally offers two major benefits:

  1. Instantaneous processing since everything is handled on the device.
  2. Enhanced privacy as there is no need to send data to the cloud for processing.

Even though there aren’t many products that allow mobile devices to run models like Llama 3.2 smoothly just yet, we can still explore it using a Linux environment on Android.


Steps to Run Llama 3.2 on Android

1. Install Termux on Android

Termux is a terminal emulator that allows Android devices to run a Linux environment without needing root access. It’s available for free and can be downloaded from the Termux GitHub page.

For this guide, download the termux-app_v0.119.0-beta.1+apt-android-7-github-debug_arm64-v8a.apk and install it on your Android device.

2. Set Up Termux

After launching Termux, follow these steps to set up the environment:

  1. Grant Storage Access:
   termux-setup-storage
Enter fullscreen mode Exit fullscreen mode

This command lets Termux access your Android device’s storage, enabling easier file management.

  1. Update Packages:
   pkg upgrade
Enter fullscreen mode Exit fullscreen mode

Enter Y when prompted to update Termux and all installed packages.

  1. Install Essential Tools:
   pkg install git cmake golang
Enter fullscreen mode Exit fullscreen mode

These packages include Git for version control, CMake for building software, and Go, the programming language in which Ollama is written.

3. Install and Compile Ollama

Ollama is a platform for running large models locally. Here’s how to install and set it up:

  1. Clone Ollama's GitHub Repository:
   git clone --depth 1 https://github.com/ollama/ollama.git
Enter fullscreen mode Exit fullscreen mode
  1. Navigate to the Ollama Directory:
   cd ollama
Enter fullscreen mode Exit fullscreen mode
  1. Generate Go Code:
   go generate ./...
Enter fullscreen mode Exit fullscreen mode
  1. Build Ollama:
   go build .
Enter fullscreen mode Exit fullscreen mode
  1. Start Ollama Server:
   ./ollama serve &
Enter fullscreen mode Exit fullscreen mode

Now the Ollama server will run in the background, allowing you to interact with the models.

4. Running Llama 3.2 Models

To run the Llama 3.2 model on your Android device, follow these steps:

  1. Choose a Model:

    • Models like llama3.2:3b (3 billion parameters) are available for testing. These models are quantized for efficiency. You can find a list of available models on Ollama’s website.
  2. Download and Run the Llama 3.2 Model:

   ./ollama run llama3.2:3b --verbose
Enter fullscreen mode Exit fullscreen mode

The --verbose flag is optional and provides detailed logs. After the download is complete, you can start interacting with the model.

5. Managing Performance

While testing Llama 3.2 on devices like the Samsung S21 Ultra, performance was smooth for the 1B model and manageable for the 3B model, though you may notice lag on older hardware. If performance is too slow, switching to the smaller 1B model can significantly improve responsiveness.


Optional Cleanup

After using Ollama, you may want to clean up the system:

  1. Remove Unnecessary Files:
   chmod -R 700 ~/go
   rm -r ~/go
Enter fullscreen mode Exit fullscreen mode
  1. Move the Ollama Binary to a Global Path:
   cp ollama/ollama /data/data/com.termux/files/usr/bin/
Enter fullscreen mode Exit fullscreen mode

Now, you can run ollama directly from the terminal.


Conclusion

Llama 3.2 represents a major leap forward in AI technology, bringing powerful, multimodal models to mobile devices. By running these models locally using Termux and Ollama, developers can explore the potential of privacy-first, on-device AI applications that don’t rely on cloud infrastructure. With models like Llama 3.2, the future of mobile AI looks bright, allowing faster, more secure AI solutions across various industries.

Reinvent your career. Join DEV.

It takes one minute and is worth it for your career.

Get started

Top comments (9)

Collapse
 
honor_soke_b6e75fff4660a profile image
Honoré SOKE

Great article. Thank

Collapse
 
b9joker108 profile image
b9Joker108

I got this error 🤕 on my Samsung Galaxy Tab S9 Ultra:

❯ go build .
# github.com/ollama/ollama/llama
ggml-quants.c:4023:88: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
ggml-quants.c:4023:76: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
ggml-quants.c:4023:64: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
ggml-quants.c:4023:52: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
# github.com/ollama/ollama/discover
gpu_info_cudart.c:61:13: warning: comparison of different enumeration types ('cudartReturn_t' (aka 'enum cudartReturn_enum') and 'enum cudaError_enum') [-Wenum-compare]
Enter fullscreen mode Exit fullscreen mode
Collapse
 
lanbase profile image
lanbase • Edited

Hi, I have got the same error on Honor Magic6 Pro (snapdragon 8 gen3).

~/ollama $ go build .
# github.com/ollama/ollama/discover
gpu_info_cudart.c:61:13: warning: comparison of different enumeration types ('cudartReturn_t' (aka 'enum cudartReturn_enum') and 'enum cudaError_enum') [-Wenum-compare]
# github.com/ollama/ollama/llama
ggml-quants.c:4023:88: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
ggml-quants.c:4023:76: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
ggml-quants.c:4023:64: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
ggml-quants.c:4023:52: error: always_inline function 'vmmlaq_s32' requires target feature 'i8mm', but would be inlined into function 'ggml_vec_dot_q4_0_q8_0' that is compiled without support for 'i8mm'
~/ollama $


Enter fullscreen mode Exit fullscreen mode

update :

I have found a workaround here:

github.com/ollama/ollama/issues/7292

cheers.

Collapse
 
jswetzen profile image
Johan • Edited

I had the same, but found a workaround here

Basically, you modify llama.go#L37-L38 to remove -D__ARM_FEATURE_MATMUL_INT8

Collapse
 
nigel_burton_686709bbab14 profile image
Nigel Burton

Qualcomm's spec sheets for Snapdragon 8 gen 3 suggest it can use GPU and a DSP to speed up LLM inference.

Do you, or any other readers know whether Ollama is taking advantage of the hardware?

if not are there any open source projects that are utilizing the full capabilities of the Gen3?

Thanks for the very useful article.

Collapse
 
learnai profile image
Learn AI

Based on the search results, here’s a detailed response to your questions regarding the utilization of Snapdragon 8 Gen 3 hardware (GPU and DSP) for LLM inference, particularly with Ollama and other open-source projects:


1. Is Ollama Taking Advantage of Snapdragon 8 Gen 3 Hardware?

As of the latest information, Ollama does not currently fully utilize the GPU and DSP capabilities of the Snapdragon 8 Gen 3 for LLM inference. While Ollama supports running models like Llama 3.2 on Android devices using Termux, its primary focus has been on CPU-based inference. There are discussions and efforts to integrate GPU and NPU support, but these are still in progress and not yet fully realized.

For example:

  • A user reported that the NPU on a Snapdragon X Elite device (which shares similar architecture with Snapdragon 8 Gen 3) was not being utilized when running Ollama.
  • Developers have mentioned that GPU support (via OpenCL) is being worked on, but NPU support will require more effort and has no estimated timeline.

2. Open-Source Projects Utilizing Snapdragon 8 Gen 3 Hardware

Several open-source projects and frameworks are actively leveraging the full capabilities of the Snapdragon 8 Gen 3, including its GPU, DSP, and NPU for AI and LLM tasks:

a. Qualcomm AI Hub Models

  • The Qualcomm AI Hub Models project provides optimized machine learning models for Snapdragon devices, including the Snapdragon 8 Gen 3. These models are designed to take advantage of the hardware's CPU, GPU, and NPU for tasks like image classification, object detection, and LLM inference.
  • The project supports various runtimes, including Qualcomm AI Engine Direct, TensorFlow Lite, and ONNX, enabling efficient deployment on Snapdragon hardware.

b. MiniCPM-Llama3-V 2.5

  • This open-source multimodal model is optimized for deployment on Snapdragon 8 Gen 3 devices. It uses 4-bit quantization and integrates with Qualcomm’s QNN framework to unlock NPU acceleration, achieving significant speed-ups in image encoding and language decoding.
  • The model demonstrates how open-source projects can leverage Snapdragon hardware for efficient on-device AI applications.

c. Llama.cpp

  • Llama.cpp is a popular open-source project for running LLMs locally. While it primarily focuses on CPU inference, there are ongoing efforts to add GPU and NPU support for Snapdragon devices. For example, developers are working on an OpenCL-based backend for Adreno GPUs, which could extend to Snapdragon 8 Gen 3.
  • Some users have reported successful performance benchmarks on Snapdragon X Elite devices, indicating potential for future optimizations.

d. Qualcomm AI Engine Direct

  • This framework allows developers to compile and optimize models for Snapdragon hardware, including the GPU and NPU. It is used in projects like EdgeStableDiffusion, which demonstrates how large models like Stable Diffusion can be efficiently run on Snapdragon 8 Gen 2 and Gen 3 devices.

3. Future Prospects

  • Ollama: While Ollama does not yet fully utilize Snapdragon 8 Gen 3 hardware, the development community is actively working on GPU and NPU support. This could significantly improve performance for on-device LLM inference in the future.
  • Open-Source Ecosystem: Projects like Qualcomm AI Hub Models, MiniCPM-Llama3-V 2.5, and Llama.cpp are leading the way in leveraging Snapdragon hardware. These efforts highlight the potential for open-source tools to fully utilize the capabilities of modern mobile chipsets.

Conclusion

Currently, Ollama does not fully utilize the GPU and DSP capabilities of the Snapdragon 8 Gen 3, but there are promising open-source projects like Qualcomm AI Hub Models, MiniCPM-Llama3-V 2.5, and Llama.cpp that are making significant strides in this area. As development continues, we can expect more tools to take full advantage of Snapdragon hardware for efficient on-device AI and LLM inference.

For further details, you can explore the referenced projects and discussions in the search results.

Collapse
 
Sloan, the sloth mascot
Comment deleted
Collapse
 
ernestoyoofi profile image
Ernestoyoofi

your url is incomplete, it should be like this https://github.com/ollama/ollama.git

Collapse
 
nann_tthein_3523dd95f8f profile image
Nann T T Hein

Thank you 🫢

This post blew up on DEV in 2020:

js visualized

🚀⚙️ JavaScript Visualized: the JavaScript Engine

As JavaScript devs, we usually don't have to deal with compilers ourselves. However, it's definitely good to know the basics of the JavaScript engine and see how it handles our human-friendly JS code, and turns it into something machines understand! 🥳

Happy coding!

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay