DEV Community: Nasser Maronie

Schema In, Data Out: A Smarter Way to Mock

Nasser Maronie — Sun, 22 Jun 2025 16:51:03 +0000

A Practical Tool for Structured JSON Data Generation

Building Software Is Like Building an Ecosystem

Every software project is like building a small, living ecosystem. You don’t just write code — you define relationships, behaviors, structures, flows, constraints. Everything affects everything else: a database model informs the API, which feeds the frontend, which influences UX decisions, which loops back into validation rules and error states.

In the middle of that web is one quiet but critical requirement: data. Every part of the stack needs it — not just eventually, but right from the start.

The moment you start prototyping, testing, or designing flows, you need something to move through the system. You need mock data. Not just for one field or one example — but full sets of realistic, structured, rule-compliant objects that look and behave like real-world inputs.

It shows up in every stage:

Designing UI with real-looking content
Testing logic that depends on field constraints
Simulating API responses during integration
Seeding local environments before backend logic exists
Demonstrating flows with actual variety, not placeholder values

But generating that data is usually annoying. Most of us either fake it manually, duplicate a handful of examples, or lean on tools that don’t understand schema structure or field-level constraints. The process works — until it breaks. And then it breaks often.

Mock Data Is Easy — Until It Isn’t

Developers, QA engineers, and data-heavy product teams all hit the same wall at some point: the need for structured, realistic mock data — fast. It’s not hard to mock a single user object. What gets frustrating is mocking multiple records of users with valid emails, localized names, structured metadata, and constraints like length, range, pattern, or nested field relationships.

MockingJar

“is a tool built from that pain.”

Not a startup pitch. Not another playground for random fake data. Just a solid web app designed to generate test data that fits your schema, follows rules, and respects structure.

It combines a visual schema builder, a powerful validation engine, and an AI-driven generation backend using the Claude API. You define the shape. You guide the content. MockingJar fills in the rest — intelligently, repeatably, and without requiring manual fixes.

Why This Exists

This project came out of repetition. Over and over, you start with:

“Let’s build a quick mock for this.”
And a few hours later you’re still writing nested JSON by hand, patching fake values, or tweaking IDs and arrays that don’t quite match the real structure.

MockingJar exists to eliminate that friction — especially in the early stages of building software. These days, engineers spend more time prototyping, validating ideas, and building POCs than ever before. With that pace, needing structured, schema-valid mock data isn’t an edge case — it’s daily life.

This tool gives you back that time — by doing one thing well: generating structured data from a schema you define.

What It Actually Does

MockingJar is split into two major components:

1. Web Interface

A full-featured web app built with:

Next.js 15 (App Router)
Material UI v7
React Hook Form + Zod for validation
Next Auth + Google OAuth for secure login
CSRF + Session + Protected routes for multiple layer of security
PostgreSQL via Prisma for data persistence
Zustand + React Query for frontend state & data sync

This interface includes:

A visual schema builder with tree view
A natural language prompt interface for AI-driven data generation
A personal schema library with save/edit/delete support
Authentication and session management
JSON preview and export tools

2. mockingjar-lib Core Engine

A standalone TypeScript package with:
A schema builder & manipulation engine
A field-level JSON validator
An Anthropic Claude integration layer
Support for nested objects, arrays, references, and complex constraints
Surgical regeneration of only invalid fields
Full test coverage
This is the same engine used in the web app — also usable in backend scripts, or test runners.

Building the Schema

MockingJar revolves around user-defined schemas. You can define field types, logic constraints, and structure down to each level.

Supported types include:

text, number, boolean, date, email, url
object (with nested fields)
array (with item constraints and nested structures)
schema (reference to another saved schema)

Each field supports logic like:

Required/optional
Min/max values or lengths
Pattern matching (e.g. ^ORD-\d{6}$)
Enum choices
Size constraints for arrays

Schema Composition

You can embed existing schemas within others. This “schema” type allows you to modularize shared models — like embedding an Address schema into multiple forms (shippingAddress, billingAddress, etc.) without redefining it every time.

Schema Builder Highlights

Auto-save system
Tree view for deep nesting
JSON-to-schema converter (paste in a JSON sample and convert to schema)
Schema selection dialog (for embedding existing schemas)
Templates for common models

Generating Data

Once the schema is ready, you can generate data with a prompt like:

“Generate order data for a customer based in London, purchasing a product related to home utilities (e.g., kitchen appliances, cleaning equipment, or home maintenance tools)”

The prompt doesn’t replace your schema — it simply guides the tone and realism of the values generated. MockingJar uses Claude Sonnet 4 behind the scenes to fill each field while respecting:

Type
Constraints
Nesting
Array lengths
Regex and enum rules

And You can configure how many records you want.

You’ll get a result preview and formatted output. If any fields fail validation, they’re retried individually — not thrown out with the rest. That recovery process is key to keeping data generation fast and consistent.

The mockingjar-lib Core Engine
This is where all the real magic lives — and you can use it directly in your Node.js projects with any Frontend preference.

Example:

import { Generator } from 'mockingjar-lib';

const result = await Generator.generate(
  'your-anthropic-api-key',
  yourSchema,
  'Generate Indonesian customer records with valid emails',
  {
    count: 10,
    enableFallback: true,
    timeout: 60000
  }
);
if (result.success) {
  console.log(result.data);
} else {
  console.error(result.errors);
}

You can also:

Validate JSON against a schema (Validation.validate)
Convert JSON into a schema (Schema.convert.jsonToSchema)
Add/update/remove deeply nested fields
Get structured metadata about generation performance

Developer-Focused Architecture

Full TypeScript codebase
ESLint + strict mode enforced
Modular codebase with generator, schema, validation, and anthropic layers
Easily extensible to add custom generators or field types

🧪 Still Evolving — Open to Collaboration

MockingJar isn’t a polished product. It’s a functional, real-world tool that solves a specific problem — and it does that well — but there’s plenty of room to grow.

Some areas still need refinement:

UI usability and form ergonomics
Schema versioning and import/export
More intuitive handling of deeply nested structures
Improved AI prompt interpretation and feedback
Community schema/template sharing
Better error visibility and debugging flows

The core generation engine (mockingjar-lib) is solid and tested, but the experience around it — especially in the web app — is still being shaped.

This project isn’t trying to be everything from day one. It’s being built piece by piece with real usage in mind, and if you’re the kind of developer who sees opportunities in that kind of foundation, your help would be welcome.

MockingJar is fully open-source. Contributions, testing, feedback, bug reports, and PRs are all appreciated — whether it’s fixing a bug, improving a prompt flow, or just pointing out where something feels off.

If this tool has solved even a small annoyance for you, and you want to help push it further, jump in.

Web App → https://mockingjar.vercel.app/mockingjar
Web App Repository → https://github.com/firstpersoncode/mockingjar
Core Engine Package → mockingjar-lib
Core Engine Repository → https://github.com/firstpersoncode/mockingjar-lib

Let’s build something useful — together.

Final Thought

MockingJar wasn’t made to impress. It was made to stop wasting time — especially on a task that no one wants to do more than once.

If you regularly write schemas, mock APIs, or build test setups, this might be a tool worth adding to your stack. And if not, steal the parts you like — that’s what open source is for.

Between Structure and Imagination: What happens when code becomes a sketchpad for ideas.

Nasser Maronie — Sat, 21 Jun 2025 19:44:16 +0000

The way I work as a software engineer has changed — and I didn’t exactly plan for it.

A few years ago, my day was mostly about implementations. Writing boilerplate, stitching things together, obsessing over the finer points of state management or input validation. But with the rise of AI tools — not just code assistants, but a growing ecosystem of intelligent helpers — something shifted. I found myself writing fewer lines of code, and instead spending more time designing, prototyping, and exploring ideas at a higher level.

Today, most of what I do is closer to rapid architecture and experimentation. I move fast — spin up MVPs, test proof of concepts, iterate, throw things away, rebuild. It’s liberating. But it introduced a new kind of bottleneck I wasn’t expecting:

Data.

Lots of DATA!

Almost everything I build needs mock data. And not just random fake names and emails — I’m talking about realistic, deeply structured, schema-specific data. Think of things like: a customer profile with conditional fields, nested address arrays, dates that follow logical patterns, optional properties, and so on. Every prototype, every API, every UI test — they all need something to work with.

At first, I didn’t think much of it. I’d just throw together a quick JSON object by hand. Then another. Then a third, slightly different. And before long, I was knee-deep in repetitive grunt work again — copying, tweaking, checking, re-checking. I was generating test data by hand more than I was writing actual features.

It hit me one night while building a prototype: I’d spent nearly an hour massaging mock data instead of solving the actual problem. And not for the first time. The frustration wasn’t even loud — just a quiet, dull hum that had been following me for years. I realized this wasn’t just a one-off pain point. This was a pattern.

I didn’t want to build another side project. I really didn’t. I had too many already, and I knew the cost of spinning up something new. But the idea wouldn’t leave me alone. What if I could define the structure of the data visually — not by writing JSON, but by shaping it intuitively — and have the machine generate the rest for me? Not magic. Not fantasy. Just something practical that worked.

So I started building it.

At first, it was rough. I stitched together a schema editor. Hooked in a basic validation system. Added a bit of natural language input for fun — something to help describe the data I wanted, not just its shape. And then I tested it. I built a schema for a customer record. Hit generate. And it worked.

I stared at the screen for a long moment. Not because it was perfect — it wasn’t — but because for once, I hadn’t had to fight the data. It felt... natural. Easy. Like something I would’ve used ten projects ago if it existed.

That little tool became what I now call:

“MockingJar”

It’s not a revolution. It’s not AI magic. It’s just a quiet little utility that solves a problem that kept getting in my way. A tool that knows how to treat structure seriously, that validates data properly, and that fills in the blanks in a way that respects what you’ve defined.

And more than anything, it’s a reminder that not all software needs to be loud or groundbreaking to matter. Sometimes, the most satisfying thing you can build is the one that makes your life 10% easier — every single day.

I didn’t mean to build it. But it wouldn’t stop bothering me until I did.

And I’m glad I did.

A Better Way to Fake Data (Without Faking It All Manually)

There’s something oddly painful about creating mock data.

It sounds simple, right? Just come up with a few fields, throw in some names and numbers, maybe a fake email or two — done. But if you've ever actually had to mock data for a real project, you know how quickly it spirals. One minute you're typing { "name": "John Doe" }, and the next you're deep in a jungle of nested arrays, conditional fields, date formats, min-max constraints, and a creeping sense that you’ve just become your own QA department.

That’s basically what led me to build MockingJar — a little tool that makes it easier to define structured JSON data and then generate mock content for it that actually makes sense.

The idea is simple: you describe the shape of the data you want — visually — and then MockingJar handles the rest. It builds out the schema, lets you customize every detail, and when you're ready, it fills it with realistic mock data.

But let me show you what that actually feels like.

You Don’t Write JSON — You Build It

Instead of starting with an empty text editor and a blinking cursor, you start with a blank canvas and a visual builder. You can add fields like "name", "age", or "email", choose their types (text, number, boolean, object, array, etc.), and define rules like whether a field is required, how many items an array should have, or what pattern a string needs to follow.

It’s kind of like building a form — except you’re designing the shape of the data your API or frontend might use later. The builder supports nesting, too, so you can go as deep as you want: objects inside arrays inside objects? No problem.

Describe What You Want, Then Hit Generate

Once your schema is ready, here’s where it gets fun: you can write a short, simple prompt like

“Generate 20 employee records with names, emails, job titles, and start dates after 2015.”

…and it will figures it out.

Behind the scenes, there’s a bit of AI involved — it reads your schema, parses your prompt, and comes up with data that fits both. The schema you defined is treated as the source of truth. So even if the AI gets creative, the result always gets checked, corrected, and cleaned up before it’s returned to you.

You end up with realistic data, shaped exactly how you wanted it, ready to copy, download, or feed directly into your app or tests.

If Something’s Off, It Fixes Just That

One of the most useful parts — and something that I honestly didn’t plan from the beginning — is how it handles mistakes. Sometimes, data doesn’t validate perfectly the first time. Maybe one field doesn’t match a regex. Maybe an array has too few items.

Instead of throwing everything out and trying again, MockingJar just regenerates the broken parts. It keeps what worked, replaces what didn’t. Like a quiet little assistant that says, “Hey, I’ve got this,” and moves on.

You don’t really notice it unless you’re paying attention — and that’s kind of the point.

Why It Feels Different

There are a lot of data generation tools out there. Some are great at generating fake names and emails. Some let you define schema in code. But they always felt disconnected to me — like I had to either do everything manually, or hand over control entirely.

MockingJar sits somewhere in the middle. You define the structure. You describe what you want. And it fills in the rest — realistically, reliably, and in a way that doesn’t fight you.

I didn’t build it to be clever. I just wanted to stop faking data by hand every time I started a new project.

Turns out, that was a pretty decent reason.

AI Didn’t Replace My Work as an Engineer — It Changed the Way I Think

I’ve seen a lot of conversations over the past few years about whether AI will replace software engineers.

And to be honest? Most of them miss the point.

AI isn’t replacing us.It’s reshaping how we work, what we focus on, and how we think about solving problems.

At least, that’s what it’s done for me.

In the past, writing software felt like a series of manual checkpoints. You’d plan a feature, build all the scaffolding, write boilerplate code, manually mock your data, wire up endpoints, and then finally, eventually, get to the “real” problem. It was a slow, linear climb.

Now? I skip half those steps without even realizing it.

The Shift From Manual to Mental

When AI tools became a real part of my workflow — autocomplete, code generation, chat-based debugging — something interesting happened. My job didn’t get easier, exactly. But it changed shape. I stopped spending energy on things I already knew how to do. Instead, I started thinking more like a problem designer than just a problem solver.

That’s a subtle but powerful shift.

I move faster now. I iterate more freely. I try ideas I would’ve skipped before because the setup would’ve taken too long. Prototyping became fun again — not because AI did it for me, but because I no longer had to sweat the small stuff to get moving.

AI Didn’t Take the Wheel — It Sat in the Passenger Seat

The more I’ve worked with AI, the more I’ve realized: it’s not here to take over. It doesn’t write perfect code. It doesn’t understand edge cases. It doesn't replace architecture or decision-making.

But what it does do — and this is huge — is amplify what I’m already good at.

It gives me momentum.It shortens the distance between idea and execution.It helps me think more clearly by letting me focus on structure, logic, and flow — not on syntax or boilerplate.

And because of that, I actually build more. I explore more. I solve better problems.

For example — a tool I built to help generate structured mock data from custom JSON schemas.

In the pre-AI era, I might have written the whole engine manually. It would’ve worked, but it would’ve been slow, rigid, and full of edge-case handling. Building a system that understands context, generates realistic sample data, and validates it against a schema would’ve been a month-long grind.

But with AI in the mix — not as the hero, but as a collaborator — I could do something better. I built the structure, the validation logic, the schema builder. Then I handed off the tedious part (inventing realistic, varied, sensible data) to the AI.

That’s the magic spot: I define the rules, and the AI plays within them.

It didn’t do the job for me.It let me do more of the job I actually care about.

The Way We Build Software Is Changing

What I’ve learned is this: AI hasn’t just changed the tools I use. It’s changed the questions I ask when I sit down to build something.

Instead of:

“How long will this take to implement?”

I now think:

“What’s the fastest way to prove if this works?”

Instead of spending time on setup, I jump straight into validation. I can test assumptions quickly. Mock APIs, fake data, generate ideas — and toss them just as fast if they don’t hold up.

This isn’t about replacing engineers.It’s about raising the floor, so we can focus on higher-level thinking. On design. On clarity. On impact.

Final Thoughts

I don’t see AI as a threat to engineering. I see it as a mirror. It shows us which parts of our work are mechanical, and which parts are irreplaceably human.

And if we use it well — as a partner, not a crutch — it makes us better.

Smarter. Faster. More creative.Not because it thinks for us, but because it frees us up to think better.

That’s the kind of progress I can get behind.

Build Your Own RAG App: A Step-by-Step Guide to Setup LLM locally using Ollama, Python, and ChromaDB

Nasser Maronie — Mon, 01 Jul 2024 05:50:03 +0000

In an era where data privacy is paramount, setting up your own local language model (LLM) provides a crucial solution for companies and individuals alike. This tutorial is designed to guide you through the process of creating a custom chatbot using Ollama, Python 3, and ChromaDB, all hosted locally on your system. Here are the key reasons why you need this tutorial:

Full Customization: Hosting your own Retrieval-Augmented Generation (RAG) application locally means you have complete control over the setup and customization. You can fine-tune the model to fit your specific needs without relying on external services.
Enhanced Privacy: By setting up your LLM model locally, you avoid the risks associated with sending sensitive data over the internet. This is especially important for companies that handle confidential information. Training your model with private data locally ensures that your data stays within your control.
Data Security: Using third-party LLM models can expose your data to potential breaches and misuse. Local deployment mitigates these risks by keeping your training data, such as PDF documents, within your secure environment.
Control Over Data Processing: When you host your own LLM, you have the ability to manage and process your data exactly how you want. This includes embedding your private data into your ChromaDB vector store, ensuring that your data processing meets your standards and requirements.
Independence from Internet Connectivity: Running your chatbot locally means you are not dependent on an internet connection. This guarantees uninterrupted service and access to your chatbot, even in offline scenarios.

This tutorial will empower you to build a robust and secure local chatbot, tailored to your needs, without compromising on privacy or control.

Retrieval-Augmented Generation (RAG)

Retrieval-Augmented Generation (RAG) is an advanced technique that combines the strengths of information retrieval and text generation to create more accurate and contextually relevant responses. Here's a breakdown of how RAG works and why it's beneficial:

What is RAG?

RAG is a hybrid model that enhances the capabilities of language models by incorporating an external knowledge base or document store. The process involves two main components:

Retrieval: In this phase, the model retrieves relevant documents or pieces of information from an external source, such as a database or a vector store, based on the input query.
Generation: The retrieved information is then used by a generative language model to produce a coherent and contextually appropriate response.

How Does RAG Work?

Query Input: The user inputs a query or question.
Document Retrieval: The system uses the query to search an external knowledge base, retrieving the most relevant documents or snippets of information.
Response Generation: The generative model processes the retrieved information, integrating it with its own knowledge to generate a detailed and accurate response.
Output: The final response, enriched with specific and relevant details from the knowledge base, is presented to the user.

Benefits of RAG

Enhanced Accuracy: By leveraging external data, RAG models can provide more precise and detailed answers, especially for domain-specific queries.
Contextual Relevance: The retrieval component ensures that the generated response is grounded in relevant and up-to-date information, improving the overall quality of the response.
Scalability: RAG systems can be easily scaled to incorporate vast amounts of data, enabling them to handle a wide range of queries and topics.
Flexibility: These models can be adapted to various domains by simply updating or expanding the external knowledge base, making them highly versatile.

Why Use RAG Locally?

Privacy and Security: Running a RAG model locally ensures that sensitive data remains secure and private, as it does not need to be sent to external servers.
Customization: You can tailor the retrieval and generation processes to suit your specific needs, including integrating proprietary data sources.
Independence: A local setup ensures that your system remains operational even without internet connectivity, providing consistent and reliable service.

By setting up a local RAG application with tools like Ollama, Python, and ChromaDB, you can enjoy the benefits of advanced language models while maintaining control over your data and customization options.

GPU

Running large language models (LLMs) like the ones used in Retrieval-Augmented Generation (RAG) requires significant computational power. One of the key components that enable efficient processing and embedding of data in these models is the Graphics Processing Unit (GPU). Here's why GPUs are essential for this task and how they impact the performance of your local LLM setup:

What is a GPU?

A GPU is a specialized processor designed to accelerate the rendering of images and videos. Unlike Central Processing Units (CPUs), which are optimized for sequential processing tasks, GPUs excel at parallel processing. This makes them particularly well-suited for the complex mathematical computations required by machine learning and deep learning models.

Why GPUs Matter for LLMs

Parallel Processing Power: GPUs can handle thousands of operations simultaneously, significantly speeding up tasks such as training and inference in LLMs. This parallelism is crucial for the heavy computational loads associated with processing large datasets and generating responses in real-time.
Efficiency in Handling Large Models: LLMs like those used in RAG require substantial memory and computational resources. GPUs are equipped with high-bandwidth memory (HBM) and multiple cores, making them capable of managing the large-scale matrix multiplications and tensor operations needed by these models.
Faster Data Embedding and Retrieval: In a local RAG setup, embedding data into a vector store like ChromaDB and retrieving relevant documents quickly is essential for performance. High-performance GPUs can accelerate these processes, ensuring that your chatbot responds promptly and accurately.
Improved Training Times: Training an LLM involves adjusting millions (or even billions) of parameters. GPUs can drastically reduce the time required for this training phase compared to CPUs, enabling more frequent updates and refinements to your model.

Choosing the Right GPU

When setting up a local LLM, the choice of GPU can significantly impact performance. Here are some factors to consider:

Memory Capacity: Larger models require more GPU memory. Look for GPUs with higher VRAM (video RAM) to accommodate extensive datasets and model parameters.
Compute Capability: The more CUDA cores a GPU has, the better it can handle parallel processing tasks. GPUs with higher compute capabilities are more efficient for deep learning tasks.
Bandwidth: Higher memory bandwidth allows for faster data transfer between the GPU and its memory, improving overall processing speed.

Examples of High-Performance GPUs for LLMs

NVIDIA RTX 3090: Known for its high VRAM (24 GB) and powerful CUDA cores, it's a popular choice for deep learning tasks.
NVIDIA A100: Designed specifically for AI and machine learning, it offers exceptional performance with large memory capacity and high compute power.
AMD Radeon Pro VII: Another strong contender, with high memory bandwidth and efficient processing capabilities.

Investing in a high-performance GPU is crucial for running LLM models locally. It ensures faster data processing, efficient model training, and quick response generation, making your local RAG application more robust and reliable. By leveraging the power of GPUs, you can fully realize the benefits of hosting your own custom chatbot, tailored to your specific needs and data privacy requirements.

Prerequisites

Before diving into the setup, ensure you have the following prerequisites in place:

Python 3: Python is a versatile programming language that you'll use to write the code for your RAG app.
ChromaDB: A vector database that will store and manage the embeddings of our data.
Ollama: To download and serve custom LLMs in our local machine.

Step 1: Install Python 3 and setup your environment

To install and setup our Python 3 environment, follow these steps:
Download and setup Python 3 on your machine.
Then make sure your Python 3 installed and run successfully:



$ python3 --version
# Python 3.11.7

Create a folder for your project, for example, local-rag:



$ mkdir local-rag
$ cd local-rag

Create a virtual environment named venv:



$ python3 -m venv venv

Activate the virtual environment:



$ source venv/bin/activate
# Windows
# venv\Scripts\activate

Step 2: Install ChromaDB and other dependencies

Install ChromaDB using pip:



$ pip install --q chromadb

Install Langchain tools to work seamlessly with your model:



$ pip install --q unstructured langchain langchain-text-splitters
$ pip install --q "unstructured[all-docs]"

Install Flask to serve your app as a HTTP service:



$ pip install --q flask

Step 3: Install Ollama

To install Ollama, follow these steps:
Head to Ollama download page, and download the installer for your operating system.
Verify your Ollama installation by running:



$ ollama --version
# ollama version is 0.1.47

Pull the LLM model you need. For example, to use the Mistral model:



$ ollama pull mistral

Pull the text embedding model. For instance, to use the Nomic Embed Text model:



$ ollama pull nomic-embed-text

Then run your Ollama models:



$ ollama serve

Build the RAG app

Now that you've set up your environment with Python, Ollama, ChromaDB and other dependencies, it's time to build your custom local RAG app. In this section, we'll walk through the hands-on Python code and provide an overview of how to structure your application.

`app.py`

This is the main Flask application file. It defines routes for embedding files to the vector database, and retrieving the response from the model.



import os
from dotenv import load_dotenv

load_dotenv()

from flask import Flask, request, jsonify
from embed import embed
from query import query
from get_vector_db import get_vector_db

TEMP_FOLDER = os.getenv('TEMP_FOLDER', './_temp')
os.makedirs(TEMP_FOLDER, exist_ok=True)

app = Flask(__name__)

@app.route('/embed', methods=['POST'])
def route_embed():
    if 'file' not in request.files:
        return jsonify({"error": "No file part"}), 400

    file = request.files['file']

    if file.filename == '':
        return jsonify({"error": "No selected file"}), 400

    embedded = embed(file)

    if embedded:
        return jsonify({"message": "File embedded successfully"}), 200

    return jsonify({"error": "File embedded unsuccessfully"}), 400

@app.route('/query', methods=['POST'])
def route_query():
    data = request.get_json()
    response = query(data.get('query'))

    if response:
        return jsonify({"message": response}), 200

    return jsonify({"error": "Something went wrong"}), 400

if __name__ == '__main__':
    app.run(host="0.0.0.0", port=8080, debug=True)

`embed.py`

This module handles the embedding process, including saving uploaded files, loading and splitting data, and adding documents to the vector database.



import os
from datetime import datetime
from werkzeug.utils import secure_filename
from langchain_community.document_loaders import UnstructuredPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from get_vector_db import get_vector_db

TEMP_FOLDER = os.getenv('TEMP_FOLDER', './_temp')

# Function to check if the uploaded file is allowed (only PDF files)
def allowed_file(filename):
    return '.' in filename and filename.rsplit('.', 1)[1].lower() in {'pdf'}

# Function to save the uploaded file to the temporary folder
def save_file(file):
    # Save the uploaded file with a secure filename and return the file path
    ct = datetime.now()
    ts = ct.timestamp()
    filename = str(ts) + "_" + secure_filename(file.filename)
    file_path = os.path.join(TEMP_FOLDER, filename)
    file.save(file_path)

    return file_path

# Function to load and split the data from the PDF file
def load_and_split_data(file_path):
    # Load the PDF file and split the data into chunks
    loader = UnstructuredPDFLoader(file_path=file_path)
    data = loader.load()
    text_splitter = RecursiveCharacterTextSplitter(chunk_size=7500, chunk_overlap=100)
    chunks = text_splitter.split_documents(data)

    return chunks

# Main function to handle the embedding process
def embed(file):
    # Check if the file is valid, save it, load and split the data, add to the database, and remove the temporary file
    if file.filename != '' and file and allowed_file(file.filename):
        file_path = save_file(file)
        chunks = load_and_split_data(file_path)
        db = get_vector_db()
        db.add_documents(chunks)
        db.persist()
        os.remove(file_path)

        return True

    return False

`query.py`

This module processes user queries by generating multiple versions of the query, retrieving relevant documents, and providing answers based on the context.



import os
from langchain_community.chat_models import ChatOllama
from langchain.prompts import ChatPromptTemplate, PromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough
from langchain.retrievers.multi_query import MultiQueryRetriever
from get_vector_db import get_vector_db

LLM_MODEL = os.getenv('LLM_MODEL', 'mistral')

# Function to get the prompt templates for generating alternative questions and answering based on context
def get_prompt():
    QUERY_PROMPT = PromptTemplate(
        input_variables=["question"],
        template="""You are an AI language model assistant. Your task is to generate five
        different versions of the given user question to retrieve relevant documents from
        a vector database. By generating multiple perspectives on the user question, your
        goal is to help the user overcome some of the limitations of the distance-based
        similarity search. Provide these alternative questions separated by newlines.
        Original question: {question}""",
    )

    template = """Answer the question based ONLY on the following context:
    {context}
    Question: {question}
    """

    prompt = ChatPromptTemplate.from_template(template)

    return QUERY_PROMPT, prompt

# Main function to handle the query process
def query(input):
    if input:
        # Initialize the language model with the specified model name
        llm = ChatOllama(model=LLM_MODEL)
        # Get the vector database instance
        db = get_vector_db()
        # Get the prompt templates
        QUERY_PROMPT, prompt = get_prompt()

        # Set up the retriever to generate multiple queries using the language model and the query prompt
        retriever = MultiQueryRetriever.from_llm(
            db.as_retriever(), 
            llm,
            prompt=QUERY_PROMPT
        )

        # Define the processing chain to retrieve context, generate the answer, and parse the output
        chain = (
            {"context": retriever, "question": RunnablePassthrough()}
            | prompt
            | llm
            | StrOutputParser()
        )

        response = chain.invoke(input)

        return response

    return None

`get_vector_db.py`

This module initializes and returns the vector database instance used for storing and retrieving document embeddings.



import os
from langchain_community.embeddings import OllamaEmbeddings
from langchain_community.vectorstores.chroma import Chroma

CHROMA_PATH = os.getenv('CHROMA_PATH', 'chroma')
COLLECTION_NAME = os.getenv('COLLECTION_NAME', 'local-rag')
TEXT_EMBEDDING_MODEL = os.getenv('TEXT_EMBEDDING_MODEL', 'nomic-embed-text')

def get_vector_db():
    embedding = OllamaEmbeddings(model=TEXT_EMBEDDING_MODEL,show_progress=True)

    db = Chroma(
        collection_name=COLLECTION_NAME,
        persist_directory=CHROMA_PATH,
        embedding_function=embedding
    )

    return db

Run your app!

Create .env file to store your environment variables:



TEMP_FOLDER = './_temp'
CHROMA_PATH = 'chroma'
COLLECTION_NAME = 'local-rag'
LLM_MODEL = 'mistral'
TEXT_EMBEDDING_MODEL = 'nomic-embed-text'

Run the app.py file to start your app server:



$ python3 app.py

Once the server is running, you can start making requests to the following endpoints:

Example command to embed a PDF file (e.g., resume.pdf): ```bash

$ curl --request POST \
--url http://localhost:8080/embed \
--header 'Content-Type: multipart/form-data' \
--form file=@/Users/nassermaronie/Documents/Nasser-resume.pdf

Response

{
"message": "File embedded successfully"
}

- Example command to ask a question to your model:
```bash


$ curl --request POST \
  --url http://localhost:8080/query \
  --header 'Content-Type: application/json' \
  --data '{ "query": "Who is Nasser?" }'

# Response
{
  "message": "Nasser Maronie is a Full Stack Developer with experience in web and mobile app development. He has worked as a Lead Full Stack Engineer at Ulventech, a Senior Full Stack Engineer at Speedoc, a Senior Frontend Engineer at Irvins, and a Software Engineer at Tokopedia. His tech stacks include Typescript, ReactJS, VueJS, React Native, NodeJS, PHP, Golang, Python, MySQL, PostgresQL, MongoDB, Redis, AWS, Firebase, and Supabase. He has a Bachelor's degree in Information System from Universitas Amikom Yogyakarta."
}

Conclusion

By following these instructions, you can effectively run and interact with your custom local RAG app using Python, Ollama, and ChromaDB, tailored to your needs. Adjust and expand the functionality as necessary to enhance the capabilities of your application.

By harnessing the capabilities of local deployment, you not only safeguard sensitive information but also optimize performance and responsiveness. Whether you're enhancing customer interactions or streamlining internal processes, a locally deployed RAG application offers flexibility and robustness to adapt and grow with your requirements.

Check the source code in this repo:

https://github.com/firstpersoncode/local-rag

Happy coding!

Building a Web Page Summarization App with Next.js, OpenAI, LangChain, and Supabase

Nasser Maronie — Sun, 23 Jun 2024 05:27:09 +0000

An app that can understand the context of any web page.

In this article, we'll show you how to create a handy web app that can summarize the content of any web page. Using Next.js for a smooth and fast web experience, LangChain for processing language, OpenAI for generating summaries, and Supabase for managing and storing vector data, we'll build a powerful tool together.

Why We're Building It

We all face information overload with so much content online. By making an app that gives quick summaries, we help people save time and stay informed. Whether you're a busy worker, a student, or just someone who wants to keep up with news and articles, this app will be a helpful tool for you.

How it's going to be

Our app will let users enter any website URL and quickly get a brief summary of the page. This means you can understand the main points of long articles, blog posts, or research papers without reading them fully.

Potential and Impact

This summarization app can be useful in many ways. It can help researchers skim through academic papers, keep news lovers updated, and more. Plus, developers can build on this app to create even more useful features.

Next.js

Next.js is a powerful and flexible React framework developed by Vercel that enables developers to build server-side rendering (SSR) and static web applications with ease. It combines the best features of React with additional capabilities to create optimized and scalable web applications.

OpenAI

The OpenAI module in Node.js provides a way to interact with OpenAI’s API, allowing developers to leverage powerful language models like GPT-3 and GPT-4. This module enables you to integrate advanced AI functionalities into your Node.js applications.

LangChain.js

LangChain is a powerful framework designed for developing applications with language models. Originally developed for Python, it has since been adapted for other languages, including Node.js. Here’s an overview of LangChain in the context of Node.js:

What is LangChain?

LangChain is a library that simplifies the creation of applications using large language models (LLMs). It provides tools to manage and integrate LLMs into your applications, handle chaining of calls to these models, and enable complex workflows with ease.

How Large Language Models (LLM) Work?

Large Language Models (LLMs) like OpenAI’s GPT-3.5 are trained on vast amounts of text data to understand and generate human-like text. They can generate responses, translate languages, and perform many other natural language processing tasks.

Supabase

Supabase is an open-source backend-as-a-service (BaaS) platform designed to help developers quickly build and deploy scalable applications. It offers a suite of tools and services that simplify database management, authentication, storage, and real-time capabilities, all built on top of PostgreSQL

Prerequisites

Before we start, make sure you have the following:

Node.js and npm installed
A Supabase account
An OpenAI account

Step 1: Setting Up Supabase

First, we need to set up a Supabase project and create the necessary tables to store our data.

Create a Supabase Project

Go to Supabase and sign up for an account.
Create a new project and make note of your Supabase URL and API key. You'll need these later.

SQL Script for Supabase

Create a new SQL query in your Supabase dashboard and run the following scripts to create the required tables and functions:

First, create an extension if it doesn’t already exist for our vector store:



create extension if not exists vector;

Next, create a table named “documents”. This table will be used to store and embed the content of web page in vector format:



create table if not exists documents (
    id bigint primary key generated always as identity,
    content text,
    metadata jsonb,
    embedding vector(1536)
);

Now, we need a function to query our embedded data:



create or replace function match_documents (
    query_embedding vector(1536),
    match_count int default null,
    filter jsonb default '{}'
) returns table (
    id bigint,
    content text,
    metadata jsonb,
    similarity float
) language plpgsql as $$
begin
    return query
    select
        id,
        content,
        metadata,
        1 - (documents.embedding <=> query_embedding) as similarity
    from documents
    where metadata @> filter
    order by documents.embedding <=> query_embedding
    limit match_count;
end;
$$;

Next, we need to set up our table for storing the web page's detail:



create table if not exists files (
    id bigint primary key generated always as identity,
    url text not null,
    created_at timestamp with time zone default timezone('utc'::text, now()) not null
);

Step 2: Setting Up OpenAI

Create OpenAI Project

Visit the OpenAI Website: Go to OpenAI's website, sign up and create new project.
Navigate to API: After logging in, navigate to the API section and create new API key. This is usually accessible from the dashboard.

Step 3: Setting Up Next.js

Create Next.js app



$ npx create-next-app summarize-page
$ cd ./summarize-page

Install the required dependencies:



npm install @langchain/community @langchain/core @langchain/openai @supabase/supabase-js langchain openai axios

Then we will install Material UI for building our interface, feel free to use other library:



npm install @mui/material @emotion/react @emotion/styled

Step 4: OpenAI and Supabase clients

Next, we need to set up the OpenAI and Supabase clients. Create a libs directory in your project and add the following files.

`src/libs/openAI.ts`

This file will configure the OpenAI client.



import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";

const openAIApiKey = process.env.OPENAI_API_KEY;

if (!openAIApiKey) throw new Error('OpenAI API Key not found.')

export const llm = new ChatOpenAI({
  openAIApiKey,
  modelName: "gpt-3.5-turbo",
  temperature: 0.9,
});

export const embeddings = new OpenAIEmbeddings(
  {
    openAIApiKey,
  },
  { maxRetries: 0 }
);

llm: The language model instance, which will generate our summaries.
embeddings: This will create embeddings for our documents, which help in finding similar content.

`src/libs/supabaseClient.ts`

This file will configure the Supabase client.



import { createClient } from "@supabase/supabase-js";

const supabaseUrl = process.env.SUPABASE_URL || "";
const supabaseAnonKey = process.env.SUPABASE_ANON_KEY || "";

if (!supabaseUrl) throw new Error("Supabase URL not found.");
if (!supabaseAnonKey) throw new Error("Supabase Anon key not found.");

export const supabaseClient = createClient(supabaseUrl, supabaseAnonKey);

supabaseClient: The Supabase client instance to interact with our Supabase database.

Step 5: Creating Services for Content and Files

Create a services directory and add the following files to handle fetching content and managing files.

`src/services/content.ts`

This service will fetch the web page content and clean it by removing HTML tags, scripts, and styles.



import axios from "axios";

export async function getContent(url: string): Promise<string> {
  let htmlContent: string = "";

  const response = await axios.get(url as string);

  htmlContent = response.data;

  if (!htmlContent) return "";

  // Remove unwanted elements and tags
  return htmlContent
    .replace(/style="[^"]*"/gi, "")
    .replace(/<style[^>]*>[\s\S]*?<\/style>/gi, "")
    .replace(/\s*on\w+="[^"]*"/gi, "")
    .replace(
      /<script(?![^>]*application\/ld\+json)[^>]*>[\s\S]*?<\/script>/gi,
      ""
    )
    .replace(/<[^>]*>/g, "")
    .replace(/\s+/g, " ");
}

This function fetches the HTML content of a given URL and cleans it up by removing styles, scripts, and HTML tags.

`src/services/file.ts`

This service will save the web page content into Supabase and retrieve summaries.



import { embeddings, llm } from "@/libs/openAI";
import { supabaseClient } from "@/libs/supabaseClient";
import { SupabaseVectorStore } from "@langchain/community/vectorstores/supabase";
import { StringOutputParser } from "@langchain/core/output_parsers";
import {
  ChatPromptTemplate,
  HumanMessagePromptTemplate,
  SystemMessagePromptTemplate,
} from "@langchain/core/prompts";
import {
  RunnablePassthrough,
  RunnableSequence,
} from "@langchain/core/runnables";
import { RecursiveCharacterTextSplitter } from "langchain/text_splitter";
import { formatDocumentsAsString } from "langchain/util/document";

export interface IFile {
  id?: number | undefined;
  url: string;
  created_at?: Date | undefined;
}

export async function saveFile(url: string, content: string): Promise<IFile> {
  const doc = await supabaseClient
    .from("files")
    .select()
    .eq("url", url)
    .single<IFile>();

  if (!doc.error && doc.data?.id) return doc.data;

  const { data, error } = await supabaseClient
    .from("files")
    .insert({ url })
    .select()
    .single<IFile>();

  if (error) throw error;

  const splitter = new RecursiveCharacterTextSplitter({
    separators: ["\n\n", "\n", " ", ""],
  });

  const output = await splitter.createDocuments([content]);
  const docs = output.map((d) => ({
    ...d,
    metadata: { ...d.metadata, file_id: data.id },
  }));

  await SupabaseVectorStore.fromDocuments(docs, embeddings, {
    client: supabaseClient,
    tableName: "documents",
    queryName: "match_documents",
  });

  return data;
}

export async function getSummarization(fileId: number): Promise<string> {
  const vectorStore = await SupabaseVectorStore.fromExistingIndex(embeddings, {
    client: supabaseClient,
    tableName: "documents",
    queryName: "match_documents",
  });

  const retriever = vectorStore.asRetriever({
    filter: (rpc) => rpc.filter("metadata->>file_id", "eq", fileId),
    k: 2,
  });

  const SYSTEM_TEMPLATE = `Use the following pieces of context, explain what is it about and summarize it.
      If you can't explain it, just say that you don't know, don't try to make up some explanation.
      ----------------
      {context}`;

  const messages = [
    SystemMessagePromptTemplate.fromTemplate(SYSTEM_TEMPLATE),
    HumanMessagePromptTemplate.fromTemplate("{format_answer}"),
  ];
  const prompt = ChatPromptTemplate.fromMessages(messages);
  const chain = RunnableSequence.from([
    {
      context: retriever.pipe(formatDocumentsAsString),
      format_answer: new RunnablePassthrough(),
    },
    prompt,
    llm,
    new StringOutputParser(),
  ]);

  const format_summarization =
    `
    Give it title, subject, description, and the conclusion of the context in this format, replace the brackets with the actual content:

    [Write the title here]

    By: [Name of the author or owner or user or publisher or writer or reporter if possible, otherwise leave it "Not Specified"]

    [Write the subject, it could be a long text, at least minimum of 300 characters]

    ----------------

    [Write the description in here, it could be a long text, at least minimum of 1000 characters]

    Conclusion:
    [Write the conclusion in here, it could be a long text, at least minimum of 500 characters]
    `;

  const summarization = await chain.invoke(format_summarization);

  return summarization;
}

saveFile: Saves the file and its content to Supabase, splits the content into manageable chunks, and stores them in the vector store.
getSummarization: Retrieves relevant documents from the vector store and generates a summary using OpenAI.

Step 6: Creating an API Handler

Now, let's create an API handler to process the content and generate a summary.

`pages/api/content.ts`



import { getContent } from "@/services/content";
import { getSummarization, saveFile } from "@/services/file";
import { NextApiRequest, NextApiResponse } from "next";

export default async function handler(
  req: NextApiRequest,
  res: NextApiResponse
) {
  if (req.method !== "POST")
    return res.status(404).json({ message: "Not found" });

  const { body } = req;

  try {
    const content = await getContent(body.url);
    const file = await saveFile(body.url, content);
    const result = await getSummarization(file.id as number);
    res.status(200).json({ result });
  } catch (err) {
    res.status(

500).json({ error: err });
  }
}

This API handler receives a URL, fetches the content, saves it to Supabase, and generates a summary. It handles both the saveFile and getSummarization functions from our services.

Step 7: Building the Frontend

Finally, let's create the frontend in src/pages/index.tsx to allow users to input URLs and display the summarizations.

`src/pages/index.tsx`



import axios from "axios";
import { useState } from "react";
import {
  Alert,
  Box,
  Button,
  Container,
  LinearProgress,
  Stack,
  TextField,
  Typography,
} from "@mui/material";

export default function Home() {
  const [loading, setLoading] = useState(false);
  const [url, setUrl] = useState("");
  const [result, setResult] = useState("");
  const [error, setError] = useState<any>(null);

  const onSubmit = async () => {
    try {
      setError(null);
      setLoading(true);
      const res = await axios.post("/api/content", { url });
      setResult(res.data.result);
    } catch (err) {
      console.error("Failed to fetch content", err);
      setError(err as any);
    } finally {
      setLoading(false);
    }
  };

  return (
    <Box sx={{ height: "100vh", overflowY: "auto" }}>
      <Container
        sx={{
          backgroundColor: (theme) => theme.palette.background.default,
          position: "sticky",
          top: 0,
          zIndex: 2,
          py: 2,
        }}
      >
        <Typography sx={{ mb: 2, fontSize: "24px" }}>
          Summarize the content of any page
        </Typography>

        <TextField
          fullWidth
          label="Input page's URL"
          value={url}
          onChange={(e) => {
            if (result) setResult("");
            setUrl(e.target.value);
          }}
          sx={{ mb: 2 }}
        />

        <Button
          disabled={loading}
          variant="contained"
          onClick={onSubmit}
        >
          Summarize
        </Button>
      </Container>

      <Container maxWidth="lg" sx={{ py: 2 }}>
        {loading ? (
          <LinearProgress />
        ) : (
          <Stack sx={{ gap: 2 }}>
            {result && (
              <Alert>
                <Typography
                  sx={{
                    whiteSpace: "pre-line",
                    wordBreak: "break-word",
                  }}
                >
                  {result}
                </Typography>
              </Alert>
            )}
            {error && <Alert severity="error">{error.message || error}</Alert>}
          </Stack>
        )}
      </Container>
    </Box>
  );
}

This React component allows users to input a URL, submit it, and display the generated summary. It handles loading states and error messages to provide a better user experience.

Step 8: Running the Application

Create a .env file in the root of your project to store your environment variables:



SUPABASE_URL=your-supabase-url
SUPABASE_ANON_KEY=your-supabase-anon-key
OPENAI_API_KEY=your-openai-api-key

Finally, start your Next.js application:



npm run dev

Now, you should have a running application where you can input web page's url, and receive the page's summarized responses.

Conclusion

Congratulations! You've built a fully functional web page summarization application using Next.js, OpenAI, LangChain, and Supabase. Users can input a URL, fetch the content, store it in Supabase, and generate a summary using OpenAI's capabilities. This setup provides a robust foundation for further enhancements and customization based on your needs.

Feel free to expand on this project by adding more features, improving the UI, or integrating additional APIs.

Check the source code in this repo:

https://github.com/firstpersoncode/summarize-page

Happy coding!

From Zero to Chatbot: How Large Language Models (LLMs) Work and How to Harness Them Easily

Nasser Maronie — Sun, 23 Jun 2024 03:11:16 +0000

With Node.js, OpenAI and a cup of coffee.

Imagine having a super smart friend who has read every book, article, and blog post on the internet. This friend can answer your questions, help you with creative writing, and even chat with you about any topic under the sun. That’s essentially what a Large Language Model (LLM) is!

Now imagine you can build one!

Large Language Models (LLMs)

Large Language Models (LLMs) like OpenAI’s GPT (Generative Pre-trained Transformer) are revolutionizing how we interact with technology. These models, trained on vast amounts of text data, can understand and generate human-like text, making them ideal for applications such as chatbots. In this article, we’ll explore the fundamentals of LLMs, the concept of prompt engineering, and how to build a chatbot using Node.js, LangChain, and OpenAI.

Key Features of LLMs:

Contextual Understanding: LLMs can understand the context of a given input, making their responses coherent and contextually relevant.
Versatility: These models can handle a wide range of tasks, including translation, summarization, and conversation.
Scalability: LLMs can be fine-tuned for specific applications, enhancing their performance for particular use cases.

Working with LLMs

To effectively utilize LLMs, it’s essential to understand how they process inputs and generate outputs. This involves crafting prompts — inputs that guide the model to produce desired responses.

Prompt Structure: A well-structured prompt provides clear instructions and sufficient context. The quality of the prompt directly influences the quality of the output.

Tokenization: LLMs process text by breaking it down into smaller units called tokens. Each token can be as short as one character or as long as one word. The model’s understanding is built on these tokens.

Temperature and Max Tokens:

Temperature: Controls the randomness of the output. Lower values make the output more deterministic, while higher values increase randomness.
Max Tokens: Limits the length of the generated response. Setting an appropriate max token value ensures responses are concise and relevant.

Prompt Engineering

Imagine you’re talking to a very knowledgeable friend who can answer any question you have. You start by asking a general question, but they respond with a clarifying question to understand exactly what you need. This back-and-forth continues until they provide you with a clear and helpful answer.

This is similar to Prompt Engineering with AI. When we interact with large language models (LLMs) like OpenAI’s GPT-3, we provide them with well-crafted prompts that give enough context for generating relevant responses.

For instance, if you ask an AI chatbot, “What are the benefits of Node.js?” and it gives a technical response, you might refine your prompt: “Can you explain the advantages of Node.js for web development?” This structured approach helps the AI understand your query and provide an accurate response.

Prompt Engineering allows developers to communicate effectively with AI, creating smart and responsive chatbots that can assist with a variety of tasks.

Prompt Engineering is the art of designing prompts to elicit specific responses from an LLM. It’s a crucial aspect of working with these models, as the prompt determines how the model interprets and responds to the input.

Tips for Effective Prompt Engineering:

Be Clear and Specific: Ensure the prompt clearly defines the task. Ambiguous prompts lead to ambiguous responses.
Provide Context: Give the model enough information to understand the context of the request.
Iterate and Refine: Experiment with different prompts and refine them based on the model’s responses.

Building a Chatbot with Node.js and LangChain

Now, let’s dive into the fun part: building a chatbot using Node.js, LangChain, and OpenAI. We’ll focus on how prompt engineering can enhance the chatbot’s responses.

Setting Up Your Environment:

Initialize a Node.js Project:

mkdir chatbot-app
cd chatbot-app
npm init -y
npm install langchain openai axios

Create the Chatbot Structure:

const { OpenAI } = require('langchain');
const axios = require('axios');

const openai = new OpenAI({
    apiKey: 'YOUR_OPENAI_API_KEY',  // Replace with your OpenAI API key
});

async function generateResponse(prompt) {
    const response = await openai.complete({
        model: 'text-davinci-003',  // You can use other models available
        prompt: prompt,
        maxTokens: 150,
    });

    return response.data.choices[0].text.trim();
}

Implementing Prompt Engineering with LangChain:

const { OpenAI, PromptTemplate } = require('langchain');

const openai = new OpenAI({
    apiKey: 'YOUR_OPENAI_API_KEY',
});

const template = new PromptTemplate({
    inputVariables: ['query'],
    template: `You are a helpful assistant. Answer the following question: {query}`
});

async function generateResponse(query) {
    const prompt = await template.format({ query });
    const response = await openai.complete({
        model: 'text-davinci-003',
        prompt: prompt,
        maxTokens: 150,
    });

    return response.data.choices[0].text.trim();
}

// Example usage
(async () => {
    const userQuery = "What are the benefits of using Node.js?";
    const response = await generateResponse(userQuery);
    console.log(response);
})();

Testing and Refining Your Chatbot

Testing is crucial to ensure your chatbot provides accurate and helpful responses. Here are some example interactions:

Basic Query:

User: “What is Node.js?”
Chatbot: “Node.js is a JavaScript runtime built on Chrome’s V8 JavaScript engine.”

Complex Query:

User: “How does asynchronous programming work in Node.js?”
Chatbot: “Asynchronous programming in Node.js allows non-blocking operations, meaning multiple tasks can be handled concurrently without waiting for previous tasks to complete.”

By iterating on the prompts and responses, you can fine-tune your chatbot to provide more accurate and helpful answers.

Conclusion

Building a chatbot with Node.js, LangChain, and OpenAI is an exciting and accessible way to harness the power of LLMs. Understanding the fundamentals of LLMs and mastering prompt engineering are key to creating a chatbot that delivers accurate and contextually relevant responses. I hope this guide inspires you to explore the potential of LLMs in your applications.

Building a Custom Chatbot with Next.js, Langchain, OpenAI, and Supabase.

Nasser Maronie — Thu, 20 Jun 2024 14:43:29 +0000

A chatbot system that can be trained with custom data from PDF files.

Nasser Maronie on LinkedIn: #nextjs #langchain #openai #chatgpt #ai #machinelearning #demo

🚀 Excited to share my latest project: a custom ChatGPT-like app built using NextJS, LangChain, and OpenAI! 🤖 In this demo, I showcase a unique feature of…

linkedin.com

In this tutorial, we will create a chatbot system that can be trained with custom data from PDF files. The chatbot will utilize Next.js for the frontend, MaterialUI for the UI components, Langchain and OpenAI for working with language models, and Supabase to store the data and embeddings. By the end, you will have a fully functional chatbot that can answer questions based on the contents of uploaded PDF files.

Next.js

OpenAI

LangChain.js

What is LangChain?

Key Features

Model Integration: Connect and interact with various language models, including those from OpenAI, Hugging Face, and more.
Prompt Management: Manage, optimize, and format prompts effectively.
Chain Building: Create sequences of model interactions, enabling more sophisticated workflows.
Memory Management: Maintain context between interactions, making the models’ responses more coherent and contextually aware.
Tool Use: Integrate external tools and APIs to augment the capabilities of language models.
Streaming: Support for streaming responses, which is useful for real-time applications.

How Large Language Models (LLM) Work?

Supabase

Our Goals

Training the Model

PDF File Conversion: The uploaded PDF files are converted into vectors. Vectors are numerical representations that the AI can understand.
Embedding: The vectors are embedded into a vector store for efficient querying.

Chatting with the Trained Model

User Input: The user provides an input query.
Prompt Conversion: The input is converted into a standalone question.
Vector Conversion: The question is converted into a vector.
Nearest Match Search: The system searches for the nearest match in the vector store.
Response Generation: The system generates an answer based on the closest match.

Prerequisites

Before we start, ensure you have the following:

Node.js and npm installed
A Supabase account
API key for OpenAI

Step 1: Setting Up Supabase

Creating Tables and Functions

First, create an extension if it doesn’t already exist for our vector store:

create extension if not exists vector;

Next, create a table named “documents”. This table will be used to store and embed the content of our uploaded PDF files in vector format:

create table if not exists documents (
    id bigint primary key generated always as identity,
    content text,
    metadata jsonb,
    embedding vector(1536)
);

Now, we need a function to query our embedded data:

create or replace function match_documents (
    query_embedding vector(1536),
    match_count int default null,
    filter jsonb default '{}'
) returns table (
    id bigint,
    content text,
    metadata jsonb,
    similarity float
) language plpgsql as $$
begin
    return query
    select
        id,
        content,
        metadata,
        1 - (documents.embedding <=> query_embedding) as similarity
    from documents
    where metadata @> filter
    order by documents.embedding <=> query_embedding
    limit match_count;
end;
$$;

The “match_documents” function performs the task of querying the embedded data. We will call this function in our Next.js app via Supabase Vector Store.

Next, we need to set up our tables for the chatbot system:

create table if not exists files (
    id bigint primary key generated always as identity,
    name text not null,
    created_at timestamp with time zone default timezone('utc'::text, now()) not null
);

create table if not exists rooms (
    id bigint primary key generated always as identity,
    created_at timestamp with time zone default timezone('utc'::text, now()) not null
);

create table if not exists chats (
    id bigint primary key generated always as identity,
    room bigint references rooms(id) on delete cascade,
    role text not null,
    message text not null,
    created_at timestamp with time zone default timezone('utc'::text, now()) not null
);

The “files” table will store details of the uploaded PDF files. This allows us to reference and filter the files in the “documents” table. Our chatbot system will query embedding data with the given “file id” selected in our app. This way, our chatbot system can manage multiple PDF files and focus on the context of a specific file.

The “rooms” table will store all the chat sessions, allowing users to have multiple chat sessions within our app.

Finally, the “chats” table will store all the chats from a particular chat session (room). The role will differentiate whether it’s a user or a bot. If it’s a user, the role will be “user”.

Step 2: Setting Up Next.js

Create Next.js app

$ npx create-next-app chatbot
$ cd ./chatbot

Install the required dependencies:

npm install @langchain/community @langchain/core @langchain/openai @supabase/supabase-js langchain openai pdf-parse pdfjs-dist

Then we will install Material UI for building our interface, feel free to use other library:

npm install @mui/material @emotion/react @emotion/styled

Connecting to Supabase

Create a file to connect your Next.js app to Supabase:

// src/libs/supabaseClient.ts

import { createClient, SupabaseClient } from "@supabase/supabase-js";

const supabaseUrl: string = process.env.NEXT_PUBLIC_SUPABASE_URL || "";
const supabaseAnonKey: string = process.env.NEXT_PUBLIC_SUPABASE_ANON_KEY || "";

if (!supabaseUrl) throw new Error("Supabase URL not found.");
if (!supabaseAnonKey) throw new Error("Supabase Anon key not found.");

export const supabaseClient: SupabaseClient = createClient(supabaseUrl, supabaseAnonKey);

Setting Up LLM clients

Create a file to set up LangChain and Embeddings:

// src/libs/openAI.ts

import { ChatOpenAI, OpenAIEmbeddings } from "@langchain/openai";

const openAIApiKey: string = process.env.NEXT_PUBLIC_OPENAI_API_KEY || "";

if (!openAIApiKey) throw new Error("OpenAI API key not found.");

export const llm = new ChatOpenAI({
  openAIApiKey,
  modelName: "gpt-3.5-turbo",
  temperature: 0.9,
});

export const embeddings = new OpenAIEmbeddings(
  {
    openAIApiKey,
  },
  { maxRetries: 0 }
);

Next.js Config

Lastly, we need to update our Next.js config file, since we will be using Web PDF Loader from Langchain, and it depends on fs module that will throw error if used in the browser. So update your config file following this snippet:

/** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: true,
  output: "export",
  webpack: (config, { isServer }) => {
    // See https://webpack.js.org/configuration/resolve/#resolvealias
    config.resolve.alias = {
      ...config.resolve.alias,
      sharp$: false,
      "onnxruntime-node$": false,
    };
    config.experiments = {
      ...config.experiments,
      topLevelAwait: true,
      asyncWebAssembly: true,
    };
    config.module.rules.push({
      test: /\.md$/i,
      use: "raw-loader",
    });
    // Fixes npm packages that depend on `fs` module
    if (!isServer) {
      config.resolve.fallback = {
        ...config.resolve.fallback, // if you miss it, all the other options in fallback, specified
        // by next.js will be dropped. Doesn't make much sense, but how it is
        fs: false, // the solution
        "node:fs/promises": false,
        module: false,
        perf_hooks: false,
      };
    }
    return config;
  },
};
export default nextConfig;

Now our Next.js app is ready! let’s continue on building the chatbot system.

Step 3: Prepare the services that communicate with our database

We will use these services / methods to communicate with the Supabase from our React component.

File Service

The file service handles file-related operations, such as fetching the list of files and saving a new file to the database.

// src/services/file.ts
import { embeddings } from "@/libs/openAI";
import { supabaseClient } from "@/libs/supabaseClient";
import { WebPDFLoader } from "@langchain/community/document_loaders/web/pdf";
import { SupabaseVectorStore } from "@langchain/community/vectorstores/supabase";

export interface IFile {
  id?: number | undefined;
  name: string;
  created_at?: Date | undefined;
}

// Fetch the list of uploaded files from the Supabase database.
export async function fetchFiles(): Promise<IFile[]> {
  const { data, error } = await supabaseClient
    .from("files")
    .select()
    .order("created_at", { ascending: false })
    .returns<IFile[]>();

  if (error) throw error;

  return data;
}

// Save a new file to the database, convert it to vectors, and store the vectors.
export async function saveFile(file: File): Promise<IFile> {
  const { data, error } = await supabaseClient
    .from("files")
    .insert({ name: file.name })
    .select()
    .single<IFile>();

  if (error) throw error;

  const loader = new WebPDFLoader(file);
  const output = await loader.load();
  const docs = output.map((d) => ({
    ...d,
    metadata: { ...d.metadata, file_id: data.id },
  }));

  await SupabaseVectorStore.fromDocuments(docs, embeddings, {
    client: supabaseClient,
    tableName: "documents",
    queryName: "match_documents",
  });

  return data;
}

fetchFiles: Fetches the list of uploaded files from the Supabase database, ordered by creation date.
saveFile: Saves a new file to the database, converts the PDF content to vectors using the Langchain library, and stores the vectors in the Supabase vector store.

Room Service

The room service handles operations related to chat rooms, such as fetching the list of rooms and creating a new room.

// src/services/room.ts
import { supabaseClient } from "@/libs/supabaseClient";

export interface IRoom {
  id?: number | undefined;
  created_at?: Date | undefined;
}

// Fetch the list of chat rooms from the Supabase database.
export async function fetchRooms(): Promise<IRoom[]> {
  const { data, error } = await supabaseClient
    .from("rooms")
    .select()
    .order("created_at", { ascending: false })
    .returns<IRoom[]>();

  if (error) throw error;

  return data;
}

// Create a new chat room in the database.
export async function createRoom(): Promise<IRoom> {
  const { data, error } = await supabaseClient
    .from("rooms")
    .insert({})
    .select()
    .single<IRoom>();

  if (error) throw error;

  return data;
}

fetchRooms: Fetches the list of chat rooms from the Supabase database, ordered by creation date.
createRoom: Creates a new chat room in the database and returns the created room.

Chat Service

The chat service handles operations related to chats, such as fetching the list of chats, posting a new chat, and getting an answer from the chatbot.

// src/services/chat.ts
import { embeddings, llm } from "@/libs/openAI";
import { supabaseClient } from "@/libs/supabaseClient";
import { SupabaseVectorStore } from "@langchain/community/vectorstores/supabase";
import { StringOutputParser } from "@langchain/core/output_parsers";
import {
  ChatPromptTemplate,
  HumanMessagePromptTemplate,
  SystemMessagePromptTemplate,
} from "@langchain/core/prompts";
import {
  RunnablePassthrough,
  RunnableSequence,
} from "@langchain/core/runnables";
import { formatDocumentsAsString } from "langchain/util/document";

export interface IChat {
  id?: number | undefined;
  room: number;
  role: string;
  message: string;
  created_at?: Date | undefined;
}

// Fetch the list of chats for a given room from the Supabase database.
export async function fetchChats(roomId: number): Promise<IChat[]> {
  const { data, error } = await supabaseClient
    .from("chats")
    .select()
    .eq("room", roomId)
    .order("created_at", { ascending: true })
    .returns<IChat[]>();

  if (error) throw error;

  return data;
}

// Post a new chat message to the database.
export async function postChat(chat: IChat): Promise<IChat> {
  const { data, error } = await supabaseClient
    .from("chats")
    .insert(chat)
    .select()
    .single<IChat>();

  if (error) throw error;

  return data;
}

// Get an answer from the chatbot based on the user's chat message.
export async function getAnswer(chat: IChat, fileId: number): Promise<IChat> {
  const vectorStore = await SupabaseVectorStore.fromExistingIndex(embeddings, {
    client: supabaseClient,
    tableName: "documents",
    queryName: "match_documents",
  });

  const retriever = vectorStore.asRetriever({
    filter: (rpc) => rpc.filter("metadata->>file_id", "eq", fileId),
    k: 2,
  });

  const SYSTEM_TEMPLATE = `Use the following pieces of context to answer the question at the end.
      If you don't know the answer, just say that you don't know, don't try to make up an answer.
      ----------------
      {context}`;

  const messages = [
    SystemMessagePromptTemplate.fromTemplate(SYSTEM_TEMPLATE),
    HumanMessagePromptTemplate.fromTemplate("{question}"),
  ];
  const prompt = ChatPromptTemplate.fromMessages(messages);
  const chain = RunnableSequence.from([
    {
      context: retriever.pipe(formatDocumentsAsString),
      question: new RunnablePassthrough(),
    },
    prompt,
    llm,
    new StringOutputParser(),
  ]);

  const answer = await chain.invoke(chat.message);

  const { data, error } = await supabaseClient
    .from("chats")
    .insert({
      role: "bot",
      room: chat.room,
      message: answer,
    })
    .select()
    .single<IChat>();

  if (error) throw error;

  return data;
}

fetchChats: Fetches the list of chats for a given room from the Supabase database, ordered by creation date.
postChat: Posts a new chat message to the database.
getAnswer: Gets an answer from the chatbot based on the user’s chat message. It uses the Langchain library to retrieve the most relevant documents from the vector store and generates a response using OpenAI’s language model.

Step 4: Building the UI

Chat Room Component

The ChatRoom component handles the display and interaction of the chat interface.

// src/components/ChatRoom.tsx

import {
  Box,
  Button,
  LinearProgress,
  Stack,
  TextField,
  Typography,
} from "@mui/material";
import { ChangeEvent, MouseEvent, useEffect, useState } from "react";
import { IChat, fetchChats, getAnswer, postChat } from "@/services/chat";

export default function ChatRoom({
  roomId,
  fileId,
}: {
  roomId: number;
  fileId: number;
}) {
  const [message, setMessage] = useState<string>("");
  const [chats, setChats] = useState<IChat[]>([]);
  const [submitting, setSubmitting] = useState(false);

  const onChangeInput = (e: ChangeEvent<HTMLInputElement>) =>
    setMessage(e.target.value);

  const onSubmitInput = async (e: MouseEvent<HTMLElement>) => {
    e.preventDefault();
    if (!message) return;
    let currChats = [...chats];
    try {
      setSubmitting(true);
      const chat = await postChat({
        role: "user",
        room: roomId,
        message,
      });

      setMessage("");
      currChats.push(chat);

      const answer = await getAnswer(chat, fileId);
      currChats.push(answer);

      setChats(currChats);
    } catch (err) {
      console.error(err);
    } finally {
      setSubmitting(false);
    }
  };

  useEffect(() => {
    (async () => {
      try {
        if (typeof roomId !== "undefined") {
          const chats = await fetchChats(roomId);
          setChats(chats);
        }
      } catch (err) {
        console.error(err);
      }
    })();
  }, [roomId]);

  return (
    <>
      <Stack sx={{ gap: 2, mb: 2 }}>
        {chats.map((chat, i) => (
          <Box
            key={i}
            sx={{
              display: "flex",
              justifyContent: chat.role === "user" ? "flex-end" : "flex-start",
            }}
          >
            <Box
              sx={{
                minWidth: "250px",
                maxWidth: "1000px",
                p: 2,
                border: "1px solid #555",
                borderRadius: (theme) => theme.spacing(2),
              }}
            >
              <Typography
                sx={{
                  whiteSpace: "pre-line",
                  wordBreak: "break-word",
                  mb: 2,
                  display: "block",
                }}
              >
                {chat.message}
              </Typography>
            </Box>
          </Box>
        ))}
      </Stack>

      {submitting && <LinearProgress />}

      <TextField
        fullWidth
        multiline
        minRows={2}
        maxRows={10}
        value={message}
        label="Write Something ..."
        onChange={onChangeInput}
        sx={{ mb: 2 }}
      />

      <Button
        fullWidth
        type="submit"
        variant="contained"
        onClick={onSubmitInput}
        disabled={submitting}
      >
        <Typography>Send</Typography>
      </Button>
    </>
  );
}

useEffect: Fetches the list of chats for the room when the component mounts or the roomId changes.
onSubmitInput: Handles sending a new chat message, posting it to the database, and getting a response from the chatbot.

File Uploader Component

The FileUploader component handles uploading files.

// src/components/Fi

import { ChangeEvent, MouseEvent, useState } from "react";
import { Box, Button, Typography } from "@mui/material";
import { IFile, saveFile } from "@/services/file";

export default function FileUploader({
  onSave,
}: {
  onSave: (file: IFile) => void;
}) {
  const [inputFile, setInputFile] = useState<File | undefined>(undefined);
  const [uploading, setUploading] = useState<boolean>(false);

  const onChangeFile = (e: ChangeEvent<HTMLInputElement>) => {
    const file = e?.target?.files?.[0];
    setInputFile(file);
  };

  const handleSaveFile = async (e: MouseEvent<HTMLElement>) => {
    e.preventDefault();
    if (!inputFile) return;
    try {
      setUploading(true);
      const file = await saveFile(inputFile);
      onSave(file);
    } catch (err) {
      console.error(err);
    } finally {
      setUploading(false);
    }
  };

  return (
    <>
      <Box
        component="label"
        htmlFor="file-uploader"
        sx={{ mb: 2, display: "block" }}
      >
        <input
          accept="application/pdf"
          id="file-uploader"
          type="file"
          style={{ display: "none" }}
          onChange={onChangeFile}
        />
        <Button variant="outlined" fullWidth component="span">
          <Typography>{inputFile ? inputFile.name : "Select File"}</Typography>
        </Button>
      </Box>

      <Button
        fullWidth
        variant="contained"
        color="primary"
        disabled={!inputFile || uploading}
        onClick={handleSaveFile}
      >
        <Typography>Upload</Typography>
      </Button>
    </>
  );
}

handleSaveFile: Handles file upload, saving the file to the database, and updating the list of files.

Home Page

The Home component is the main page that allows the user to create or select chat room, upload a file, and selecting a file to chat about.

// src/pages/index.tsx

import ChatRoom from "@/components/ChatRoom";
import FileUploader from "@/components/FileUploader";
import { IFile, fetchFiles } from "@/services/file";
import { IRoom, createRoom, fetchRooms } from "@/services/room";
import {
  Button,
  Divider,
  Grid,
  List,
  ListItemButton,
  Typography,
} from "@mui/material";
import { MouseEvent, useEffect, useMemo, useState } from "react";

export default function Home() {
  const [rooms, setRooms] = useState<IRoom[]>([]);
  const [files, setFiles] = useState<IFile[]>([]);
  const [roomId, setRoomId] = useState<number | undefined>(undefined);
  const [fileId, setFileId] = useState<number | undefined>(undefined);

  const onSaveFile = (file: IFile) => setFiles((v) => [file, ...v]);

  const handleCreateRoom = async (e: MouseEvent<HTMLElement>) => {
    e.preventDefault();
    try {
      const newRoom = await createRoom();
      setRooms((v) => [newRoom, ...v]);
      setRoomId(newRoom.id);
    } catch (err) {
      console.error(err);
    }
  };

  const handleSelectRoom =
    (id: number | undefined) => (e: MouseEvent<HTMLElement>) => {
      e.preventDefault();
      setRoomId(id);
    };

  const handleSelectFile =
    (id: number | undefined) => (e: MouseEvent<HTMLElement>) => {
      e.preventDefault();
      setFileId(id);
    };

  useEffect(() => {
    (async () => {
      try {
        const rooms = await fetchRooms();
        setRooms(rooms);

        const files = await fetchFiles();
        setFiles(files);
      } catch (err) {
        console.error(err);
      }
    })();
  }, []);

  return (
    <Grid container>
      <Grid item xs={2} sx={{ p: 2 }}>
        <Button fullWidth variant="contained" onClick={handleCreateRoom}>
          New Chat
        </Button>
        <Divider sx={{ my: 2 }} />
        <List>
          {rooms.map((room, i) => (
            <ListItemButton
              selected={roomId === room.id}
              key={i}
              onClick={handleSelectRoom(room.id)}
            >
              {room.created_at?.toString()}
            </ListItemButton>
          ))}
        </List>
      </Grid>

      <Grid item xs={2} sx={{ p: 2 }}>
        <FileUploader onSave={onSaveFile} />
        <Divider sx={{ my: 2 }} />
        <List>
          {files.map((file, i) => (
            <ListItemButton
              selected={fileId === file.id}
              key={i}
              onClick={handleSelectFile(file.id)}
            >
              {file.name}
            </ListItemButton>
          ))}
        </List>
      </Grid>

      <Grid item xs sx={{ p: 2 }}>
        {roomId && fileId ? (
          <ChatRoom roomId={roomId as number} fileId={fileId as number} />
        ) : (
          <Typography>Select one room and one file</Typography>
        )}
      </Grid>
    </Grid>
  );
}

onSaveFile: Callback to becalled once our FileUploader component save the file into database successfully, this way we can update the “files” state with the new file.
handleCreateRoom: Handles on creating a new chat room
handleSelectRoom: Handles selecting a room
handleSelectFile: Handles selecting a file.
Conditional Rendering: Renders the text helper component if no file and room is selected, and the ChatRoom component if a file and room is selected.

Step 5: Running the Application

Create a .env file in the root of your project to store your environment variables:

NEXT_PUBLIC_SUPABASE_URL=your-supabase-url
NEXT_PUBLIC_SUPABASE_ANON_KEY=your-supabase-anon-key
NEXT_PUBLIC_OPENAI_API_KEY=your-openai-api-key

Finally, start your Next.js application:

npm run dev

Now, you should have a running application where you can upload PDF files, chat with a bot trained on your data, and receive relevant responses based on the uploaded content.

Conclusion

This guide provided a comprehensive overview of building a custom chatbot that can answer questions based on uploaded PDF files. You learned how to set up your project, configure Supabase and OpenAI, create the necessary services, and build the frontend components with React and MaterialUI. With this foundation, you can extend and customize the chatbot to fit your specific needs.

Check the source code in this repo:

https://github.com/firstpersoncode/chatbot

Happy coding!