Running Gemma 2B on Edge Hardware with Actian VectorAI DB

Today, running a powerful language model entirely on edge hardware is no longer the hard part. The problem is making it useful once it is there. A developer can deploy Gemma 2B on NVIDIA Jetson Orin Nano or a custom hardware device, run inference locally, and generate responses without sending a single token to the cloud.

The model performs well, supports an 8K token context window, and delivers enough throughput for many production edge applications. The real challenge appears when the agent needs to answer questions from a 50,000-document maintenance corpus that is far larger than anything that fits inside its context window.

A local model without a local vector store faces two choices. Either overload the context window with documents that do not fit or call a remote retrieval service that may not exist in an offline environment. Neither option works for industrial systems, robotics platforms, field service devices, or other regulated industries where connectivity is unreliable or unavailable.

This article builds a complete local inference stack using the Gemma 2B model. Gemma 2B handles text generation through Ollama. Actian VectorAI DB handles semantic retrieval from a large document corpus. The entire stack runs on the device, requires no cloud services during operation, and provides a practical foundation for retrieval-augmented agents deployed at the edge.

What Gemma 2B Brings to Edge Hardware

Most edge AI developers no longer struggle to run a language model locally. The challenge is finding one that delivers useful reasoning and text generation while fitting within the memory and power constraints of edge devices.

Gemma 2B is a lightweight open model designed for local inference. It can run on devices such as the NVIDIA Jetson Orin Nano, Raspberry Pi 5, and industrial gateways. In this setup, the model runs on-device, so data stays local instead of flowing through cloud infrastructure.

Model: Gemma 2B
Parameters: 2 billion
Disk size: ~1.6GB
Target hardware: Raspberry Pi 5, Jetson Orin Nano, edge gateways, embedded Linux devices
RAM required: 8GB+

For many edge workloads, these hardware requirements are modest enough to leave room for additional services on the same device.

Gemma 2B also supports an 8K token context window. While that is sufficient for conversations, instructions, and small document sets, it quickly becomes a limitation when an agent must search across thousands of maintenance manuals, support articles, operational procedures, or historical records. A single industrial knowledge base can contain millions of words, far exceeding what the model can hold in context at once. The model is released under a commercially friendly license, which matters if you need to deploy it on-premises or at the edge.

The problem is that local inference only solves half of the architecture. Once the corpus grows beyond what fits in the context window, the agent needs a retrieval layer that can find the right information before generation begins.

The Retrieval Gap

Getting Gemma 2B running locally solves the generation problem. It does not solve the retrieval problem. An edge agent can only answer questions using information that exists in its prompt or context window. As soon as the knowledge base grows beyond what fits in memory, the agent needs a way to locate the right information before generating a response.

This approach has some failure modes.

Context window overflow

An 8K token context window sounds large until the agent must work with a real-world document collection. A library of regulatory documentation can contain hundreds of thousands or even millions of words.

Without retrieval, developers often attempt to load as many documents as possible into the prompt. That approach quickly reaches the context limit. The model then truncates documents, loses important details, or generates answers from incomplete information. The result is lower accuracy and less reliable responses.

Cloud dependency

Many Retrieval-Augmented Generation (RAG) tutorials solve the context problem by connecting the model to a cloud-hosted vector database. That works in environments where internet connectivity is not a core requirement. It becomes a problem when the application runs in a highly regulated or an air-gapped environment, with no internet access.

When the retrieval layer depends on a cloud service, the agent depends on the network. If the connection fails, the retrieval fails. In many cases, the agent cannot answer questions because the information it needs never reaches the model.

Scale degradation with basic embedding search

Simple embedding search implementations work well with a few hundred documents. Many developers start with an in-memory vector index or a basic search capability bundled with a local inference framework. Performance is acceptable during testing because the dataset is small.

The situation changes when the corpus grows to tens of thousands of document chunks and the application begins serving multiple queries per second. Search latency increases, memory usage rises, and retrieval becomes the bottleneck in the system.

An edge-compliant vector store deployed on the device resolves all three challenges without adding cloud dependency.

The Full Stack

The goal is to keep both inference and retrieval on the same device. Gemma 2B handles generation while VectorAI DB handles retrieval. Together, they form a complete RAG stack that operates without cloud dependency.

In this architecture, the user never interacts with the language model directly. Every query first passes through the retrieval layer, which searches the local document corpus and returns the most relevant information. The application injects those retrieved documents into the prompt before sending it to Gemma 2B. The model then generates a response grounded in the retrieved context rather than relying solely on its training data.

The stack consists of two core components:

Gemma 2B

Gemma 2B serves as the generation layer. It receives the user query along with the retrieved context and produces the final response. Running through Ollama simplifies deployment on edge hardware by providing a lightweight local API for inference.

Responsibilities:

• Accept user prompts
• Process retrieved context
• Generate grounded responses
• Run entirely on-device

VectorAI DB

VectorAI DB serves as the retrieval layer. It stores document chunks and their metadata, then performs semantic search against the local corpus. When a user submits a query, the application converts that query into an embedding and sends it to VectorAI DB. The database returns the most semantically relevant document chunks, which become context for the language model.

Responsibilities:

• Store document embeddings
• Perform a semantic similarity search
• Retrieve relevant context
• Operate without network connectivity

This architecture keeps every component inside the device boundary. Once the model, embeddings, and database are installed, the system can answer questions without relying on external APIs, cloud-hosted vector databases, or internet connectivity.

Figure 1: Local inference and retrieval stack architecture

Setting Up the Stack

This section builds a complete local retrieval and inference stack on an edge device using Gemma 2B for inference and VectorAI DB for retrieval.

Stage 1: Install Ollama and pull Gemma 2B

Install Ollama on your custom hardware device by following the installation instructions for your operating system.

After installation, pull the Gemma 2B model. Use the latest available tag for Gemma 2B in Ollama. Confirm the exact tag in the Ollama model registry before production use.

ollama pull gemma2:2b

You should see an output:

Figure 2: Ollama pull for Gemma 2B

Verify the installation by running the test prompt

ollama run gemma2:2b "Explain what a vector database does in one sentence."

You get the result:

Figure 3: Verify Ollama Gemma 2B installation

Stage 2: Install Docker and run VectorAI DB

Install Docker by following the guide and confirm that the Docker service is running. Then, create a docker-compose.yml file with the following configuration:

services:
 vectorai:
   image: actian/vectorai:latest
   container_name: vectorai_db
   ports:
    - "6573:6573" #rest
    - "6574:6574" #grpc
   volumes:
     # vector data persists across restarts
    - ./data:/var/lib/actian-vectorai
   environment:
    - VECTORAI_LOG_LEVEL=info
    - ACTIAN_VECTORAI_ACCEPT_EULA=YES
   restart: unless-stopped

Start the VectorAI DB server by:

docker-compose up -d

Stage 3: Connect with VectorAIClient and create a collection

Install UV and run the command:

uv init .

Install the dependencies by running the command:

uv add actian-vectorai-client requests

Create a file create_collection.py with the following contents:

import requests
from actian_vectorai import VectorAIClient, VectorParams, Distance
# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_URL = "localhost:6574"
COLLECTION   = "docs"
OLLAMA_URL   = "http://localhost:11434/api/embeddings"
OLLAMA_MODEL = "gemma2:2b"
# ──────────────────────────────────────────────────────────────────────────────
def get_embedding_dim() -> int:
   """Probe Ollama once to get the embedding dimension for this model."""
   response = requests.post(
       OLLAMA_URL,
       json={"model": OLLAMA_MODEL, "prompt": "probe"},
       timeout=30,
   )
   response.raise_for_status()
   return len(response.json()["embedding"])
def main():
   print(f"Probing embedding dimension from Ollama ({OLLAMA_MODEL})...")
   dim = get_embedding_dim()
   print(f"  Embedding dimension: {dim}")
   with VectorAIClient(VECTORAI_URL) as client:
       info = client.health_check()
       print(f"\nConnected to {info['title']} v{info['version']}")


       existing = client.collections.list()
       if COLLECTION in existing:
           print(f"\nCollection '{COLLECTION}' already exists — nothing to do.")
           return
       client.collections.create(
           COLLECTION,
           vectors_config=VectorParams(size=dim, distance=Distance.Cosine),
       )
       # 5. Confirm using get_info()
       info = client.collections.get_info(COLLECTION)
       print(f"\nCollection created: '{COLLECTION}'")
       print(f"  Vector size : {dim}")
       print(f"  Distance    : Cosine")
       print(f"  Status      : {info.status}")
       print(f"  Points      : {info.points_count}")
       print(f"\nPayload fields written at ingest time:")
       print(f"  content     — chunk text")
       print(f"  source      — source filename")
       print(f"  chunk_index — position of this chunk within the document")
       print(f"  metadata    — dict with char_start and total_chunks")
if __name__ == "__main__":
   main()

create_collection.py does the following:

• Detects embedding size automatically by sending a sample text to Ollama, avoiding hardcoded vector dimensions
• Connects to VectorAI DB via gRPC and checks that the database server is running
• Checks for an existing docs collection and exits if it already exists, making the script safe to rerun
• Creates the docs collection only when it does not exist.

Run this file by:

uv run create_collection.py    

You get the following output:

Figure 4: Run create_collection.py

Stage 4: Ingest a document corpus

With the collection created, the next step is to fill it. The ingestion script reads every .txt file in a local docs/directory, splits each file into chunks, generates an embedding for each chunk using Ollama, and writes the result to VectorAI DB. No external API is called at any point in this process.
Create a docs/ directory and add a file named corpus.txt inside it with the following content:

Hydraulic Press Unit 7 — Maintenance Reference


Unit 7 is a 50-ton hydraulic press manufactured by Duratek in 2019.
Serial number: DT-7740-B. Located in Bay 3, Building 2.


Scheduled maintenance is every 90 days. Last service was completed on March 14, 2026 by technician R. Okafor.


Hydraulic fluid: ISO 46 mineral oil. Tank capacity is 12 liters.
Replace fluid every 180 days or if colour turns dark brown.


The main pump runs at 210 bar operating pressure. Maximum rated pressure is 250 bar.
If pressure drops below 180 bar during operation, inspect the pump seals.


Hydraulic coupling torque spec: 42 Nm ± 2 Nm. Use thread-lock grade 243 after torquing.


Known issue: the pressure relief valve on Unit 7 sticks occasionally at cold start.
Workaround is to run the press unloaded for 2 minutes before applying load.
Replacement valve part number: DT-PRV-114.


Emergency stop is the red panel on the left side of the frame.
Do not operate the press without the safety guard in place.

Create a file ingest.py with the following content:

import uuid
from pathlib import Path


import requests
from actian_vectorai import VectorAIClient, PointStruct
# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_URL = "localhost:6574"
COLLECTION   = "docs"
OLLAMA_URL   = "http://localhost:11434/api/embeddings"
OLLAMA_MODEL = "gemma2:2b"
DOCS_DIR     = Path("./docs")
CHUNK_SIZE   = 400    # characters per chunk
# ──────────────────────────────────────────────────────────────────────────────
def embed(text: str) -> list[float]:
   response = requests.post(
       OLLAMA_URL,
       json={"model": OLLAMA_MODEL, "prompt": text},
       timeout=60,
   )
   response.raise_for_status()
   return response.json()["embedding"]
def chunk_text(text: str) -> list[str]:
   return [text[i : i + CHUNK_SIZE] for i in range(0, len(text), CHUNK_SIZE)]
def main():
   files = list(DOCS_DIR.glob("*.txt"))
   if not files:
       print(f"No .txt files found in {DOCS_DIR}/")
       print("Create the folder and drop some .txt files in it, then re-run.")
       return
   print(f"Found {len(files)} file(s) in {DOCS_DIR}/\n")
   with VectorAIClient(VECTORAI_URL) as client:
       # Confirm the collection is there before doing any work
       existing = client.collections.list()
       if COLLECTION not in existing:
           print(f"Collection '{COLLECTION}' not found.")
           print("Run 'python create_collection.py' first.")
           return
       total_chunks = 0
       for path in files:
           text = path.read_text(encoding="utf-8")
           chunks = chunk_text(text)
           points = []
           print(f"Ingesting {path.name} ({len(chunks)} chunks)...")
           for idx, chunk in enumerate(chunks):
               vector = embed(chunk)
               points.append(
                   PointStruct(
                       id=str(uuid.uuid4()),
                       vector=vector,
                       payload={
                           "content":     chunk,
                           "source":      path.name,
                           "chunk_index": idx,
                           "metadata": {
                               "char_start":   idx * CHUNK_SIZE,
                               "total_chunks": len(chunks),
                           },
                       },
                   )
               )
           client.points.upsert(COLLECTION, points)
           total_chunks += len(points)
           print(f"  ✓ {len(points)} chunks written")
       # Final count from the DB
       count = client.points.count(COLLECTION)
       print(f"\nDone. {total_chunks} chunks ingested this run.")
       print(f"Total points in '{COLLECTION}': {count}")
if __name__ == "__main__":
   main()

The ingest.py script is responsible for preparing documents for retrieval. It performs the following steps:

1. Loads documents from the configured source directory
2. Splits each document into 400-character chunks to create focused passages that can be retrieved accurately
3. Generates embeddings for each chunk by sending the text to Ollama's /api/embeddings endpoint
4. Builds a payload containing the chunk text and associated metadata, such as the document ID, chunk ID, and source information
5. Stores the vector and payload in the vector database, allowing the chunks to be searched later using semantic similarity

After ingestion is complete, every chunk is represented by both its embedding vector and its metadata. This enables the retrieval system to find the most relevant passages for a query and return the original text along with information about where it came from.

Run the ingest.py by:

uv run ingest.py

This returns the following results:

Figure 5: Run ingest.py

Building the Retrieval Loop

The retrieval loop is the core of the system. It connects three local components into one flow:

• The user query input
• VectorAI DB for semantic retrieval
• Gemma 2B via Ollama for response generation

The full pipeline runs entirely on-device and turns a raw question into a grounded answer using retrieved context.

Create a file query.py with the following content:

import sys
import requests
from actian_vectorai import VectorAIClient


# ── Config ─────────────────────────────────────────────────────────────────────
VECTORAI_URL = "localhost:6574"
COLLECTION   = "docs"
OLLAMA_URL   = "http://localhost:11434"
OLLAMA_MODEL = "gemma2:2b"
TOP_K        = 3
# ──────────────────────────────────────────────────────────────────────────────




# Step 1: Accepts a user query
def main(query: str):
   print(f"\nQuery: {query}\n")


   # Step 2: Embed the query using the same model used at ingest time.
   # Using the same model is critical — the query vector must exist in the
   # same vector space as the stored chunk vectors for similarity to be meaningful.
   query_vector = embed(query)


   # Step 3: Search VectorAI DB for the top-k most similar chunks
   with VectorAIClient(VECTORAI_URL) as client:
       results = client.points.search(
           COLLECTION,
           vector=query_vector,
           limit=TOP_K,
       )


   if not results:
       print("No results returned. Make sure you have run ingest.py first.")
       return


   # Print retrieved chunks so retrieval is visible, not just implied
   print(f"── Retrieved {len(results)} chunk(s) ───────────────────────────")
   context_parts = []
   for i, result in enumerate(results, 1):
       source  = result.payload.get("source", "unknown")
       chunk   = result.payload.get("chunk_index", "?")
       content = result.payload.get("content", "")
       print(f"[{i}] {source} / chunk {chunk}  (score: {result.score:.3f})")
       print(f"    {content[:120]}...")
       context_parts.append(f"[{source}]\n{content}")
   print("────────────────────────────────────────────────────\n")


   # Step 4: Inject retrieved chunks into the prompt as context.
   # The model is explicitly told to use only the provided context.
   # This keeps the response grounded in the indexed documents and prevents
   # the model from drawing on its general training knowledge.
   context = "\n\n".join(context_parts)
   prompt = (
       "You are a maintenance assistant. "
       "Use only the context below to answer the question. "
       "If the answer is not in the context, say so.\n\n"
       f"Context:\n{context}\n\n"
       f"Question: {query}\n\n"
       "Answer:"
   )


   # Step 5: Call Gemma 2B via the Ollama API and return the response
   print("Generating answer with Gemma 2B...")
   response = requests.post(
       f"{OLLAMA_URL}/api/generate",
       json={"model": OLLAMA_MODEL, "prompt": prompt, "stream": False},
       timeout=120,
   )
   response.raise_for_status()
   answer = response.json()["response"].strip()
   print(f"\nAnswer:\n{answer}\n")




def embed(text: str) -> list[float]:
   response = requests.post(
       f"{OLLAMA_URL}/api/embeddings",
       json={"model": OLLAMA_MODEL, "prompt": text},
       timeout=60,
   )
   response.raise_for_status()
   return response.json()["embedding"]




if __name__ == "__main__":
   if len(sys.argv) < 2:
       print("Usage: python query.py \"your question here\"")
       sys.exit(0)
   main(" ".join(sys.argv[1:]))

This code does the following:

1. Accepts a user query: The script starts by taking a question from the command line.
2. Converts the query into an embedding: The embed() function sends the query to Ollama’s embeddings endpoint.
3. Searches VectorAI DB for relevant context: The embedding is used to query the local vector database.
4. Prints retrieved chunks: Before generating a response, the script prints the retrieved results.
5. Builds a grounded prompt for Gemma 2B: All retrieved chunks are merged into a single context block.
6. Sends the prompt to Gemma 2B via Ollama: The final step sends the structured prompt to the local model.
7. Returns the final answer: The response is extracted and printed.

Test the end-to-end flow by running the command:

uv run query.py "what is the torque spec for the hydraulic coupling?"

You get the result:

Figure 6: Building the retrieval loop

From the output, you see that the model is no longer answering from its internal training data alone. Instead, it is explicitly grounded in the retrieved chunks printed from VectorAI DB before generation. Each response is tied to specific documents, which means you can trace every answer back to the source material.

Running Offline

This is the final validation of the entire stack. At this point, both Gemma 2B and VectorAI DB are already installed, the corpus has been indexed, and the retrieval loop is working on a live connection. The next step is to prove that nothing in the runtime path depends on the internet.

Disable your network interface by running the command:

networksetup -setairportpower en0 off  # for MacOs
sudo ip link set eth0 down  # for Linux

Run the command:

uv run query.py "what is the serial number"

We get the following results:

Figure 7: Verify offline execution

Wrapping Up

This system demonstrates that edge AI is no longer limited to isolated model inference. Gemma 2B handles local text generation efficiently on devices like the Jetson Orin Nano, while VectorAI DB provides a persistent semantic memory layer that runs on the same hardware. Together, they form a complete retrieval-augmented generation stack that operates without cloud services during runtime.

After initial setup, the stack continues to function without network access. The model does not require external APIs, and the vector database does not depend on cloud infrastructure. This makes the architecture suitable for industrial environments, robotics systems, and field devices where connectivity is unreliable or restricted.

Get started with Actian VectorAI DB Community Edition by signing up today. Check the documentation for deployment and usage instructions, and participate in the Discord community for support and discussions.