Ollama & LangChain.js: Build Local, Powerful AI Apps

I created a new website: Free Access to the 8 Volumes on Typescript & AI Masterclass, no registration required. Choose Volume and chapter on the menu on the left. 160 Chapters and hundreds of quizzes at the end of chapters.

Bridging Local Intelligence with Structured Workflows

The integration of Ollama with LangChain.js represents a significant shift in how we build intelligent applications. It moves us away from relying solely on cloud-based LLM APIs and towards a modular, locally-hosted ecosystem. This approach empowers developers to create more private, performant, and deterministic AI solutions. This post will dive into the core concepts, analogies, and practical code examples to help you understand and implement this powerful combination.

Understanding the Core Concepts: Raw vs. Structured Models

At its heart, the difference lies in how we interact with the Large Language Model (LLM). Direct API integration with Ollama, while functional, is akin to using low-level sockets – sending raw strings and receiving raw strings. This lacks structure, type safety, and the ability to chain operations seamlessly.

LangChain.js acts as the "Express.js" or "Next.js" framework for LLMs, providing the necessary abstraction layer. It wraps raw model calls in standardized interfaces (LLM, Embeddings, Runnable), enabling the creation of deterministic pipelines instead of imperative scripts.

Specifically, we'll focus on two key components:

• OllamaLLM (The Reasoner): Handles text generation, taking a prompt and returning a completion. Crucially, it's a "Runnable" that can be composed into larger chains.
• OllamaEmbeddings (The Vectorizer): Transforms text into numerical vectors (embeddings) capturing semantic meaning, allowing for similarity comparisons.

The Web Development Analogy: A Full-Stack Application

Think of a LangChain.js application like a modern full-stack web app:

• Database vs. Embedding Model (OllamaEmbeddings): A database stores data, but raw data is often unstructured. We index it with search engines like Elasticsearch or vector databases like Pinecone for efficient searching. OllamaEmbeddings is the "build process" that generates these indexes (vectors) from your text, ensuring your data stays local.
• API Route vs. LLM (OllamaLLM): An API route receives requests, processes logic, and returns responses. OllamaLLM is the serverless function or API endpoint, accepting a prompt and generating a response, but within a structured, orchestratable framework.
• Orchestrator vs. Chain: Tools like Redux or server-side middleware manage data flow in complex web apps. A LangChain Chain is the middleware pipeline, connecting retrieval to generation: Retrieve -> Format Prompt -> Generate -> Parse Output.

Why Integrate Ollama with LangChain.js?

Instead of raw API calls, why wrap Ollama in LangChain.js? The benefits are significant:

• Latency: Cloud LLMs involve network requests. With OllamaLLM locally, inference happens on the same machine, drastically reducing response times.
• Privacy: Data never leaves your machine, crucial for handling sensitive information (PII).
• Structured Output: LangChain's Output Parsers enforce type safety, preventing crashes from malformed LLM responses.

Diving into the Code: A Basic Example

Let's build a semantic search engine for a product catalog using Ollama and LangChain.js.

Prerequisites:

1. Ollama installed and running locally.
2. nomic-embed-text model pulled: ollama pull nomic-embed-text.
3. Node.js environment with langchain and @langchain/community installed.

// src/ollama-langchain-basic.ts

import { OllamaEmbeddings } from "@langchain/community/embeddings/ollama";
import { Ollama } from "@langchain/community/llms/ollama";
import { MemoryVectorStore } from "langchain/vectorstores/memory";
import { Document } from "langchain/document";

const embeddings = new OllamaEmbeddings({
  model: "nomic-embed-text",
  baseUrl: "http://localhost:11434",
});

const llm = new Ollama({
  model: "llama3.1",
  baseUrl: "http://localhost:11434",
});

const documents: Document[] = [
  new Document({
    pageContent: "The QuickStart Pro running shoes feature advanced cushioning technology for long-distance runners.",
    metadata: { category: "Footwear", price: 120, id: "prod_001" },
  }),
  // ... more documents
];

async function searchProducts(query: string): Promise<Document[]> {
  const vectorStore = await MemoryVectorStore.fromDocuments(
    documents,
    embeddings
  );
  const searchResults = await vectorStore.similaritySearch(query, 2);
  return searchResults;
}

async function generateResponse(query: string, context: Document[]) {
  const contextText = context
    .map((doc) => `Product: ${doc.pageContent} (Price: $${doc.metadata.price})`)
    .join("\n");

  const prompt = `
    User Question: ${query}

    Context Products:
    ${contextText}

    Based on the context above, recommend the best product for the user.
    Be concise and friendly.
  `;
  const response = await llm.invoke(prompt);
  return response;
}

async function main() {
  const userQuery = "I need shoes for running marathons";
  const relevantDocs = await searchProducts(userQuery);
  const aiResponse = await generateResponse(userQuery, relevantDocs);
  console.log("\n=== FINAL RESULT ===");
  console.log("Recommended Product:", aiResponse);
}

main();

This example demonstrates the core workflow: embedding the query, performing a similarity search, and generating a response using the retrieved context.

Advanced Application: Semantic Search with pgvector and Ollama

For production environments, consider using pgvector with Supabase for scalable semantic search. This architecture combines the benefits of local LLM inference with a robust vector database. The code would involve:

1. Server Action: Receiving the user query.
2. Embedding Generation: Using OllamaEmbeddings to create a vector representation of the query.
3. pgvector Search: Querying the documents table in PostgreSQL using pgvector to find similar documents.
4. Context Injection: Passing the retrieved documents to OllamaLLM for a context-aware response.

Performance Optimization & Common Pitfalls

Integrating Ollama with LangChain.js requires attention to performance:

• Memory Management: Be mindful of the keep_alive parameter in Ollama to free up resources.
• Concurrency: Node.js is single-threaded. Offload heavy processing using RunnableLambda or asynchronous operations.
• Token Limits: Local models have smaller context windows. Use splitters and limit retrieved chunks.
• Ollama Timeouts: Increase timeouts for CPU-heavy models.
• JSON Parsing: Handle potential errors when parsing LLM output.

Conclusion: The Future of Local AI

By integrating Ollama with LangChain.js, you're building a local, distributed system that offers privacy, performance, and control. This approach unlocks a new era of AI development, empowering you to create powerful applications entirely on your own infrastructure. Embrace the power of local inference and structured workflows to build the next generation of intelligent applications.

The concepts and code demonstrated here are drawn directly from the comprehensive roadmap laid out in the book The Edge of AI. Local LLMs (Ollama), Transformers.js, WebGPU, and Performance Optimization Amazon Link of the AI with JavaScript & TypeScript Series.
The ebook is also on Leanpub.com: https://leanpub.com/EdgeOfAIJavaScriptTypeScript.

👉 Free Access now to the TypeScript & AI Series on Programming Central, it includes 8 Volumes, 160 Chapters and hundreds of quizzes for every chapter.