Running Llama Models Locally with Docker

Rashi Dashore — Thu, 25 Jun 2026 16:15:56 +0000

I've been experimenting with running large language models entirely on my own machine, and the setup turned out to be simpler than I expected. Here's exactly what I did to get Llama 3 running locally using Docker - no cloud API, no data leaving my machine.

Why Local Inference?

The first thing I noticed after switching to local inference was the privacy gain. Every prompt I send stays on my machine. For projects involving sensitive data, internal documents, customer queries, proprietary code, that matters. There's no third-party logging, no rate limits, and no per-token cost.

Beyond privacy, running models locally gives you full control over the model version, the inference parameters, and the runtime environment. Cloud APIs abstract all of that away. Whenever tweak temperature or context length is needed for a specific task, I could do it directly without navigating a provider dashboard. Local inference also means your application keeps working even when an external API goes down — a real advantage in production workflows.

Minimum System Requirements

Before starting, make sure your machine meets these minimums:

RAM: 8 GB minimum (16 GB recommended for smooth performance)
Disk: At least 10 GB free storage
CPU: Modern multi-core processor (x86_64 or ARM64)
GPU: Optional but significantly speeds up inference; NVIDIA with CUDA or Apple Silicon with Metal both work
OS: Linux, macOS, or Windows with WSL2

Working?

Ollama is a lightweight runtime that handles model loading, quantization, and serving over a local HTTP API. Wrapping it in Docker makes the setup portable and isolated — the model files, config, and server all live inside a named volume, separate from your system. Docker also means you can spin this up on any machine with a single command, no manual dependency installs.

Set up

I have used Ollama inside Docker, which packages the model runtime cleanly. Created a docker-compose.yml to make the setup reproducible:

# docker-compose.yml
version: "3.8"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    ports:
      - "11434:11434"
    volumes:
      - ollama_data:/root/.ollama

volumes:
  ollama_data:

Then I pulled and ran Llama 3:

docker exec -it ollama ollama pull llama3
docker exec -it ollama ollama run llama3

I added a simple Python client to query it programmatically:

import requests

response = requests.post("http://localhost:11434/api/generate", json={
    "model": "llama3",
    "prompt": "Summarize the key risks in this contract clause: ...",
    "stream": False
})
print(response.json()["response"])

Example Prompts:

"Explain this Python stack trace and suggest a fix."
"Summarize this 500-word document in three bullet points."

Response latency on the 8B model was 2–4 seconds per query — fast enough for interactive use.

Resource Quick Reference

Model	RAM Required	Disk Space
Llama 3 8B	~6 GB	~4.7 GB
Llama 3 70B	~48 GB	~40 GB

Running Llama locally with Docker took me under 15 minutes to configure, and it's now part of my standard dev environment for any task where keeping data private is non-negotiable.

Have you tried running llama models locally? How was your experience?

Becoming an AWS Cloud Captain: My Experience

Rashi Dashore — Wed, 16 Oct 2024 19:13:19 +0000

Hello AWSome people!

I’m Rashi Dashore, AWS Cloud Captain from Indore, India. In this blog, I’ll be sharing everything about the AWS Cloud Club program and my experience! If you have any questions or feedback, feel free to ping me.

Before I share my experiences, let me be very clear: the AWS Cloud Club Program is entirely focused on supporting students and creating a strong community. You’ll never find a captain selling AWS or asking you to pay for any personal benefit.

I first heard about the “AWS Cloud Clubs” when the applications for Cohort 3 opened in a LinkedIn post by Jen Looper. I was unsure about the program but was always interested in volunteering, giving back to the community, and, of course, sharing my thoughts.

The application process was as follows:

Essay: You have to answer some essay questions and share your leadership experience, AWS knowledge, and your willingness to start a club and help students at your university. My pro tip is to avoid the use of AI—write your genuine thoughts and be honest!
Video submission: You’ll be asked to submit a video explaining an AWS service. Make sure to be confident while explaining, and try to avoid directly reading from the presentation.
(2025 Update) From the upcoming Cohort 4: You’ll need to submit a document from your university confirming that they’ll support and allow an AWS Cloud Club, with a signature from a faculty member. Refer to this document if you want to explain the program to your faculty and administrators and show how it supports students.

I applied for the program, but after reading how difficult it is to become a captain due to the limited number of selections each year, I was unsure about my application. The results came on March 8, 2024, at 2 AM IST, and it was a selection email!!! I was so happy to become one of the 87 globally selected AWS Cloud Captains and got the chance to lead an AWS Cloud Club at my university.

As an AWS Cloud Captain, you not only lead a club but also take on the responsibility of helping students with AWS. Captains organize events, sessions, webinars, camps, labs, and so much more to support students! This not only helps you sharpen your technical skills but also helps you become a professional leader, connect with so many AWSome people, and get direct guidance from the AWS Advocacy Team. And who can forget the amazing AWS PERKS and SWAGS? You get perks like certification vouchers, resume reviews, invitations to monthly calls, and more!

If you ask me, I can definitely say that becoming an AWS Cloud Club Captain was one of the best things that happened to me in 2024. It not only helped me become more confident but also opened the doors to many opportunities. Unlike other ambassador programs where you constantly have to talk about the product, AWS Cloud Clubs are more about community and creating an inclusive space for students to explore the field of cloud computing.

A lot of students ask me how to become an AWS Cloud Captain. My only tip to them is to be honest and show the willingness to help the community. This program values a love for community—if you have that, this role is for you!

Because of my passion for community, I had the opportunity to organize AWS Student Community Day 2024 in Indore, which was the very first record-breaking SCD globally with 700+ attendees. It also became the biggest cloud festival Indore has ever seen. One thing I learned from this program is that despite all the odds, stress, and hurdles, if you have people who support and help you, you can achieve great things. Take a sneak peek at my pictures from AWS events with Jen Looper, Stephen Howell, Chirag Oswal, Ramneek Kalra and Sambit Bhattarai !

I didn’t intend to write this much but got carried away. If you still haven’t joined my community, what are you waiting for? Join now to learn about AWS and grab some amazing swag: https://www.meetup.com/aws-cloud-club-iips/

If you have any questions, suggestions, or just want to say hello, ping me on LinkedIn: RASHI DASHORE!
Until next time!

~ Rashi Dashore

DEV Community: Rashi Dashore