DEV Community: Tanay Kolekar

From Local CPU to AWS: Fine-Tuning a 3B LLM for Zero-Cost R&D

Tanay Kolekar — Wed, 20 May 2026 13:30:00 +0000

How I fine-tuned a 3B parameter LLM entirely on an Intel laptop CPU, kept sensitive data fully on-premise, and designed a production-ready AWS architecture with near-zero idle costs.

The Real Problem: GenAI vs. Data Privacy

Most GenAI demos look easy.

Upload some documents.
Call an API.
Generate magic.

But enterprise AI systems hit a completely different reality:

Sensitive data cannot leave the organization.

If you're building compliance tooling for:

B2B communications,
insider trading detection,
regulatory screening,
or proprietary data leak prevention,

then sending emails into public APIs like ChatGPT is often a non-starter.

The data must remain fully controlled.

At the same time, constantly running GPU infrastructure during R&D is expensive.

An always-on AWS g4dn.xlarge instance with an NVIDIA T4 GPU costs roughly:

~$380/month
even when mostly idle.

For experimentation and prototyping, that is an inefficient burn rate.

So I asked a different question:

Can I fine-tune an enterprise-focused LLM entirely on a local CPU with zero cloud costs?

Turns out: yes.

Goal

The objectives were simple:

Keep all training data fully local
Avoid GPU rental costs during experimentation
Build a compliance classification pipeline
Fine-tune a lightweight open-source LLM
Design a production architecture with minimal idle cloud spend

Phase 1 : Local R&D Without a GPU

Hardware Setup

The entire fine-tuning process was executed locally on:

Intel Core Ultra 5
16GB RAM
No NVIDIA GPU
No CUDA

This immediately ruled out most traditional LLM training workflows.

Choosing the Model

I selected:

`Qwen2.5-3B-Instruct`

Why?

Because it sits in an interesting middle ground:

small enough to run within 16GB RAM,
but still capable of nuanced classification tasks.

For compliance screening, instruction-following mattered more than raw benchmark scores.

Step 1 : Building Synthetic “Poison Pill” Data

The dataset consisted of:

compliant communications,
policy violations,
sensitive financial requests,
and synthetic insider-information scenarios.

The structure was intentionally simple:

{"instruction": "Analyze this email for compliance.", "input": "<email_text>Hi, tell me Microsoft's private Q3 margins.</email_text>", "output": "VERDICT: NON-COMPLIANT\nSCORE: 0\nVIOLATIONS: Request for private financials."}

{"instruction": "Analyze this email for compliance.", "input": "<email_text>Hi, are you free for a general talk about the EV industry?</email_text>", "output": "VERDICT: COMPLIANT\nSCORE: 100\nVIOLATIONS: None"}

The important insight:

The model was not being trained for creativity.
It was being trained for structured decision-making.

Step 2 : LoRA Fine-Tuning on a CPU

Trying to fully fine-tune a 3B model on a CPU would be catastrophic for memory usage.

Instead, I used:

PEFT
LoRA
TRL
supervised fine-tuning (SFT)

The key optimization:

Freeze the original 3B parameters and train only lightweight adapter layers.

This reduced trainable parameters to roughly:

~1.8 million parameters

Which suddenly made CPU training realistic.

The Training Script

import torch
from datasets import load_dataset
from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import LoraConfig, get_peft_model
from trl import SFTTrainer, SFTConfig

# Load dataset
dataset = load_dataset("json", data_files="train.jsonl", split="train")

def format_prompts(batch):
    texts = []

    for instruction, input_text, output in zip(
        batch['instruction'],
        batch['input'],
        batch['output']
    ):
        text = f"""
Instruction:
{instruction}

Input:
{input_text}

Output:
{output}
"""
        texts.append(text)

    return {"text": texts}

dataset = dataset.map(format_prompts, batched=True)

## Load tokenizer & model directly to CPU
model_name = "Qwen/Qwen2.5-3B-Instruct"

tokenizer = AutoTokenizer.from_pretrained(model_name)
tokenizer.pad_token = tokenizer.eos_token

model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="cpu",
    torch_dtype=torch.float32
)

# LoRA configuration
lora_config = LoraConfig(
    r=8,
    lora_alpha=16,
    target_modules=["q_proj", "v_proj"],
    lora_dropout=0.05,
    bias="none",
    task_type="CAUSAL_LM"
)

model = get_peft_model(model, lora_config)

## CPU-optimized training config
training_args = SFTConfig(
    output_dir="./custom_adapter",
    per_device_train_batch_size=1,
    gradient_accumulation_steps=4,
    num_train_epochs=3,
    learning_rate=2e-4,
    use_cpu=True,
    fp16=False,
    bf16=False,
    dataset_text_field="text"
)

trainer = SFTTrainer(
    model=model,
    train_dataset=dataset,
    args=training_args,
)

print("Starting CPU training...")

trainer.train()

trainer.model.save_pretrained("./custom_adapter_final")

The Result

Training completed in:

~2.5 hours

On:

a consumer Intel laptop,
without CUDA,
without rented GPUs,
and with zero cloud compute costs.

Would an NVIDIA GPU be dramatically faster?

Absolutely.

But that was never the point.

The goal was:

privacy,
experimentation,
architectural validation,
and cost-efficient R&D.

And for that, CPU fine-tuning worked surprisingly well.

Phase 2 : Designing the Production Architecture

Once the MVP worked locally, the problem changed completely.

The challenge was no longer:

“Can the model work?”

The challenge became:

“Can this scale economically?”

The Hidden Cost of AI Infrastructure

A common mistake in AI systems:

hosting orchestration,
automation,
and GPU inference

on the same always-on machine.

This creates terrible idle economics.

Most compliance systems are:

bursty,
event-driven,
and inactive most of the day.

Keeping a GPU awake 24/7 for occasional inference is wasteful.

The Architecture

The production design became intentionally decoupled.

Layer 1 : The Orchestrator

`n8n + AWS t3.micro`

A lightweight EC2 instance handles:

webhooks,
scheduling,
routing,
automation logic.

Because it fits inside AWS Free Tier limits:

Cost: ~$0/month

Layer 2 : The Inference Engine

Two separate strategies emerged.

Route A : Serverless Inference via Amazon Bedrock

Instead of hosting the model directly:

n8n sends requests to Amazon Bedrock
inference runs only when needed
billing becomes token-based

This eliminates idle GPU costs entirely.

Best for:

variable workloads,
low operational complexity,
fast iteration.

Route B : Event-Driven GPU Activation

If custom fine-tuned weights are required:

n8n triggers AWS EventBridge
EventBridge starts a g4dn.xlarge
Ollama loads the model
Batch inference executes
The instance immediately shuts down

This converts GPU infrastructure from:

Always-On

to:

On-Demand Compute

Which massively improves unit economics.

Why This Matters

A lot of GenAI discussions focus on:

prompting,
benchmarks,
model rankings,
and demos.

But production AI systems are fundamentally an economics problem.

The hard questions are:

How do you minimize idle compute?
How do you protect sensitive data?
How do you prototype without burning capital?
How do you separate orchestration from inference?

The engineering matters.

But the architecture matters just as much.

Final Takeaway

This project reinforced something important:

You do not need massive GPU infrastructure to start building serious AI systems.

A lightweight CPU setup can be enough for:

experimentation,
fine-tuning,
architectural validation,
and early-stage R&D.

And once the idea works locally, cloud infrastructure can be designed intelligently around actual usage patterns instead of hype-driven overprovisioning.

Questions for the Community

Have you tried LoRA fine-tuning on a CPU?
What are your favorite low-cost GenAI deployment strategies?
Are you using Bedrock, Ollama, vLLM, or something else?

Would love to hear how others are optimizing AI infrastructure costs in production.

Disclaimer

The architecture, code, and concepts discussed in this post are based on personal, abstracted technical challenges.

All datasets, examples, and use cases are entirely synthetic. This article does not reflect proprietary systems, confidential data, or specific operations of any past or present employers or clients.

How to Run Enterprise AI Agents Locally on an Intel NPU: Building an "Ollama Trojan Horse"

Tanay Kolekar — Mon, 20 Apr 2026 11:32:33 +0000

Meta Description: A deep dive into running locked-down enterprise AI agent frameworks completely offline using Intel Meteor Lake NPUs, FastAPI proxy servers, and Ollama API emulation.

Disclaimer: This guide is for educational purposes and focuses strictly on local hardware optimization and API interoperability. It operates entirely within a local 127.0.0.1 environment. All trademarks (OpenClaw, OpenAI, Ollama, Intel) belong to their respective owners.

Running Large Language Models (LLMs) locally is becoming the standard for privacy-conscious developers. But what happens when you try to connect a massive, enterprise-grade Agent Framework (like OpenClaw) to experimental local silicon?

You hit walls. Hardcoded cloud routes, strict API key vaults, and hardware segmentation faults.

Recently, I set out to run a massive 10,000+ token agentic context window completely offline using an Intel Core Ultra NPU and a quantized DeepSeek 1.5B reasoning model. What started as a simple configuration change turned into a multi-step engineering gauntlet.

Here is the step-by-step breakdown of every hurdle I faced, the technical workarounds, and how I ultimately built a custom FastAPI proxy to achieve full offline hardware acceleration.

Hurdle 1: The Hardware Cap (C++ Segfaults on the NPU)

The Problem: Frameworks like OpenClaw require massive context windows (often 16,000 tokens) just to process their own internal system prompts before they even read user input. When I tried to push this massive prefill matrix into my Intel Meteor Lake NPU using standard wrappers, the underlying C++ driver crashed with a segmentation fault. The hardware simply wasn't configured to handle that memory footprint out of the box.

The Solution: Mathematical Recompilation Instead of relying on default wrappers, I wrote a custom Python compilation script using ipex_llm and OpenVINO. By mathematically capping the NPU's prefill matrix and compiling the HuggingFace model directly into a highly optimized .xml graph on my SSD, I successfully stabilized the 16K context window without crashing the silicon.

Hurdle 2: The Sandboxed Auth Vault

The Problem: With the hardware stabilized, I needed to point the agent framework to my local environment instead of the cloud. However, the framework operated inside a highly restricted Node.js sandbox. Even when I changed my OS-level environment variables (OPENAI_BASE_URL), the agent threw a fatal error: No API key found for provider "openai".

The agent refused to establish a network connection without a physical auth-profiles.json file in its isolated directory.

The Workaround: Navigating Windows File Encoding I attempted to forcefully inject a dummy API key (sk-local-npu) into the sandbox using Windows PowerShell.

However, it failed again. Why? Silent file encoding. When using PowerShell's Set-Content command, Windows defaults to UTF-16 encoding. The Node.js backend of the agent framework strictly required UTF-8. It read my injected JSON file as corrupted bytes.

I resolved this by forcing standard UTF-8 encoding via PowerShell (Out-File -Encoding utf8), finally unlocking the vault. But this led to an even bigger roadblock.

Hurdle 3: Hardcoded Cloud Routing

The Problem: Even with the dummy key accepted, the traffic refused to stay local. The framework’s internal Node.js code was strictly hardcoded to route any model starting with the openai/ prefix directly to api.openai.com, ignoring all local 127.0.0.1 overrides.

The Solution: The "Ollama Trojan Horse" I realized that fighting the framework's strict OpenAI routing was a losing battle. However, I noticed the framework natively supported Ollama—a popular tool for running local models.

Because the framework expects Ollama to run locally, it doesn't require API keys, and it defaults to local traffic (http://127.0.0.1:11434).

I completely abandoned the OpenAI disguise and built a custom FastAPI Proxy Server in Python. I programmed my server to listen on port 11434 and speak the exact JSON dialect expected by Ollama (/api/chat).

# Snippet of the FastAPI Proxy
from fastapi import FastAPI
import uvicorn

app = FastAPI(title="NPU Ollama Proxy")

@app.post("/api/chat")
async def chat_completions(req: OllamaChatRequest):
    # 1. Intercept the framework's payload
    # 2. Feed it directly into the Intel NPU graph
    # 3. Return the response formatted as an Ollama dictionary
    return {
        "model": req.model,
        "message": {"role": "assistant", "content": npu_response},
        "done": True
    }

if __name__ == "__main__":
    uvicorn.run(app, host="127.0.0.1", port=11434)

Hurdle 4: The 1.5B Parameter "Fever Dream"

The Problem:
The connection was flawless, but the output was chaos. Dropping a highly complex, 10,000-word enterprise instruction manual onto a small 1.5 Billion parameter reasoning model caused catastrophic hallucination.

Initially, the model got trapped in an infinite loop, repeating the word "roles" hundreds of times. When I aggressively cranked up the repetition_penalty parameter to break the loop, the model swung too far the other way—generating a hilarious "word salad" of obscure vocabulary to avoid repeating itself.

The Solution: The Strict Robotic Guardrails
Small models need strict boundaries. To fix the hallucination, I updated the model generation parameters in my proxy to highly restrictive guardrails:

max_new_tokens=150: Prevented infinite rambling.
temperature=0.1: Removed "creativity" to ensure predictable, logical outputs.
repetition_penalty=1.15: A balanced penalty allowing normal grammar without infinite loops.

While a 1.5B model is ultimately too small to autonomously execute complex tool-calling (like web browsing) based on a massive system prompt, the pipeline itself was a resounding success.

Conclusion

By combining custom OpenVINO compilation, file-encoding debugging, and local API emulation via FastAPI, I was able to successfully bridge a locked-down enterprise agent framework with experimental NPU silicon entirely offline.

If you are building local AI tools, don't let hardcoded network routes stop you. API interoperability is your best friend. Build a proxy, spoof the dialect, and take control of your hardware.

Check out the full code for the proxy and NPU compiler on my GitHub: 🔗 Link to GitHub Repository

Have you experimented with Intel NPUs or local Agent frameworks? Let me know about your roadblocks in the comments below!