DEV Community: Utsav D

I Built a Tiny Neural Network From Scratch in Python — No PyTorch

Utsav D — Tue, 29 Sep 2026 19:20:00 +0000

I Built a Tiny Neural Network From Scratch in Python — No PyTorch

I didn't want to just call model.fit() and say I understood neural networks.

So I decided to build one myself.

No PyTorch.
No TensorFlow.
No Keras.

Just Python + NumPy + the mathematics behind a neural network.

The goal wasn't to build a production-ready deep learning framework. The goal was to understand what actually happens inside a neural network during training.

By the end, we will have a small neural network that can learn a binary classification problem using:

Forward propagation
ReLU activation
Sigmoid activation
Binary cross-entropy loss
Backpropagation
Gradient descent
NumPy matrix operations

1. What Are We Actually Building?

Our network will have three parts:

Input Layer
    ↓
Hidden Layer
    ↓
Output Layer

For this example:

3 input features
      ↓
4 hidden neurons
      ↓
1 output neuron

The output represents the probability of belonging to class 1.

For example:

0.02 → Class 0
0.91 → Class 1

2. The Mathematics Behind a Neuron

A neuron first calculates a weighted sum:

$$
z = XW + b
$$

where:

X = input
W = weights
b = bias
z = pre-activation value

Then an activation function is applied.

For the hidden layer we'll use ReLU:

$$
ReLU(x) = \max(0,x)
$$

For the output layer we'll use sigmoid:

$$
\sigma(x) = \frac{1}{1+e^{-x}}
$$

The sigmoid converts the output into a value between 0 and 1.

3. Creating Some Data

We'll create a small synthetic binary classification dataset.

import numpy as np

np.random.seed(42)

n_samples = 200

X = np.random.randn(n_samples, 3)

y = (
    (X[:, 0] + X[:, 1] - X[:, 2]) > 0
).astype(int).reshape(-1, 1)

print(X.shape)
print(y.shape)

The shapes are:

X → (200, 3)
y → (200, 1)

So every sample contains three features.

4. Activation Functions

Let's implement ReLU and sigmoid ourselves.

def relu(x):
    return np.maximum(0, x)


def relu_derivative(x):
    return (x > 0).astype(float)


def sigmoid(x):
    return 1 / (1 + np.exp(-x))

The derivative of ReLU is important during backpropagation.

It is:

1 when x > 0
0 when x <= 0

5. Building the Neural Network

Now we can create the actual network.

class NeuralNetwork:

    def __init__(self, input_size, hidden_size, output_size):

        self.W1 = np.random.randn(input_size, hidden_size) * 0.1
        self.b1 = np.zeros((1, hidden_size))

        self.W2 = np.random.randn(hidden_size, output_size) * 0.1
        self.b2 = np.zeros((1, output_size))

The architecture is:

3 → 4 → 1

Therefore:

W1 = 3 × 4
b1 = 1 × 4

W2 = 4 × 1
b2 = 1 × 1

6. Forward Propagation

Now we need to pass the input through the network.

def forward(self, X):

    self.z1 = X @ self.W1 + self.b1
    self.a1 = relu(self.z1)

    self.z2 = self.a1 @ self.W2 + self.b2
    self.a2 = sigmoid(self.z2)

    return self.a2

Mathematically:

$$
Z_1 = XW_1+b_1
$$

$$
A_1 = ReLU(Z_1)
$$

$$
Z_2 = A_1W_2+b_2
$$

$$
A_2 = Sigmoid(Z_2)
$$

And A2 becomes our prediction.

7. Measuring the Error

A model needs to know how wrong its predictions are.

For binary classification, we'll use binary cross-entropy:

$$
L = -\frac{1}{m}\sum
[y\log(\hat y)+(1-y)\log(1-\hat y)]
$$

In Python:

def binary_cross_entropy(y, y_pred):

    epsilon = 1e-8

    y_pred = np.clip(
        y_pred,
        epsilon,
        1 - epsilon
    )

    return -np.mean(
        y * np.log(y_pred)
        + (1 - y) * np.log(1 - y_pred)
    )

The clip() prevents numerical problems caused by taking the logarithm of zero.

8. The Part That Usually Gets Hidden: Backpropagation

This is where things get interesting.

During backpropagation, we calculate how much each parameter contributed to the error.

For the output layer:

$$
dZ_2 = A_2-y
$$

Then:

$$
dW_2 = \frac{A_1^T dZ_2}{m}
$$

$$
db_2 = \frac{\sum dZ_2}{m}
$$

Then we propagate the gradient back into the hidden layer:

$$
dA_1 = dZ_2W_2^T
$$

$$
dZ_1 = dA_1 \odot ReLU'(Z_1)
$$

Finally:

$$
dW_1 = \frac{X^TdZ_1}{m}
$$

and:

$$
db_1 = \frac{\sum dZ_1}{m}
$$

The implementation:

def backward(self, X, y, learning_rate=0.01):

    m = X.shape[0]

    # Output layer gradient
    dz2 = self.a2 - y

    dw2 = (self.a1.T @ dz2) / m
    db2 = np.sum(dz2, axis=0, keepdims=True) / m

    # Hidden layer gradient
    da1 = dz2 @ self.W2.T

    dz1 = da1 * relu_derivative(self.z1)

    dw1 = (X.T @ dz1) / m
    db1 = np.sum(dz1, axis=0, keepdims=True) / m

    # Update parameters
    self.W2 -= learning_rate * dw2
    self.b2 -= learning_rate * db2

    self.W1 -= learning_rate * dw1
    self.b1 -= learning_rate * db1

That's essentially the core of training.

9. Training the Network

Now let's put everything together.

model = NeuralNetwork(
    input_size=3,
    hidden_size=4,
    output_size=1
)

epochs = 2000
learning_rate = 0.05

loss_history = []

for epoch in range(epochs):

    # Forward pass
    predictions = model.forward(X)

    # Calculate loss
    loss = binary_cross_entropy(
        y,
        predictions
    )

    # Backpropagation
    model.backward(
        X,
        y,
        learning_rate
    )

    loss_history.append(loss)

    if epoch % 200 == 0:
        print(
            f"Epoch {epoch}, Loss: {loss:.4f}"
        )

At the beginning, the network has essentially no idea what it is doing.

Its weights are randomly initialized.

But every iteration does this:

Input
  ↓
Prediction
  ↓
Calculate Error
  ↓
Calculate Gradients
  ↓
Update Weights
  ↓
Repeat

Over time, the loss should decrease.

10. Making Predictions

After training, we can use the network to predict new samples.

predictions = model.forward(X)

predicted_classes = (
    predictions >= 0.5
).astype(int)

Let's calculate accuracy:

accuracy = np.mean(
    predicted_classes == y
)

print(
    f"Accuracy: {accuracy * 100:.2f}%"
)

The important part isn't the exact number.

The important part is that we built the learning process ourselves.

11. Looking at the Training Process

We can visualize the loss:

import matplotlib.pyplot as plt

plt.plot(loss_history)

plt.xlabel("Epoch")
plt.ylabel("Loss")
plt.title("Training Loss")

plt.show()

You should see the loss generally decrease as training progresses.

That curve is one of the simplest ways to see learning happening.

12. What Is Actually Happening During One Training Step?

It can be reduced to seven steps:

Step 1 — Input

Step 2 — Forward propagation

X → Hidden Layer → Output

Step 3 — Prediction

ŷ

Step 4 — Loss

ŷ compared with y

Step 5 — Backpropagation

Calculate gradients.

Step 6 — Gradient descent

Update the weights:

$$
W := W-\eta\frac{\partial L}{\partial W}
$$

where η is the learning rate.

Step 7 — Repeat

Thousands of times.

That's training.

13. So What Does PyTorch Actually Do?

After building this manually, frameworks like PyTorch become much easier to understand.

When you write:

loss.backward()

PyTorch is performing the gradient calculations for you.

When you write:

optimizer.step()

the parameters are updated.

And when you define:

nn.Linear(3, 4)

you're creating something conceptually similar to:

W = np.random.randn(3, 4)
b = np.zeros((1, 4))

Of course, real frameworks handle much more:

Automatic differentiation
GPU acceleration
Optimizers
Memory management
Tensor operations
Neural-network modules
Mixed precision
Distributed training

But the fundamental ideas are still the same.

14. Why Build One From Scratch?

You might be wondering:

Why spend time implementing something that PyTorch already does?

Because using a framework and understanding the underlying process are different things.

When you only use:

model.fit(...)

it's easy to think of training as a black box.

Building the network manually forces you to understand:

Why weights exist
Why biases exist
What activation functions do
What a gradient represents
Why the loss changes
How errors move backward
Why learning rate matters
Why matrix dimensions matter

Once those concepts click, high-level deep-learning code becomes much less mysterious.

15. What I Would Add Next

This implementation is intentionally tiny.

A more serious version could add:

Mini-batch gradient descent
Multiple hidden layers
Softmax for multiclass classification
Adam optimizer
Dropout
Batch normalization
L2 regularization
Model saving/loading
Train/validation/test splits
Hyperparameter tuning

The next interesting experiment would be to implement the same architecture twice:

NumPy implementation
        vs
PyTorch implementation

Then compare their training behavior and code complexity.

16. Project Structure

A simple project could look like this:

tiny-neural-network/
│
├── neural_network.py
├── train.py
├── data.py
├── visualize.py
├── requirements.txt
└── README.md

requirements.txt could be as simple as:

numpy
matplotlib

That's it.

No deep-learning framework is required.

17. The Biggest Thing I Learned

The most useful part of this experiment wasn't the final classifier.

It was realizing how much abstraction modern frameworks provide.

A few lines of PyTorch can represent operations that involve:

Matrix multiplication
        ↓
Activation
        ↓
Loss
        ↓
Derivatives
        ↓
Gradient calculation
        ↓
Parameter updates

When you understand those operations individually, frameworks stop feeling like magic.

They become tools.

Final Takeaway

You don't need to build every machine-learning model from scratch.

In fact, for real projects, you usually shouldn't.

PyTorch and TensorFlow exist for very good reasons.

But building a tiny neural network once is an excellent way to understand what happens underneath the APIs.

The next time you see:

loss.backward()
optimizer.step()

you'll have a much better idea of what those two lines actually represent.

The framework is the abstraction.
The mathematics is what makes it work.

I Made an LLM Read My PDFs Without Fine-Tuning It

Utsav D — Tue, 29 Sep 2026 19:16:26 +0000

I Built a PDF Chatbot Without Fine-Tuning an LLM — Here's How It Works

I had a simple problem.

I had a PDF containing a lot of information, and I wanted to ask questions about it.

Something like:

"What are the main findings?"

"What dataset was used?"

"Explain the methodology in simple terms."

"Where does the paper discuss its limitations?"

My first thought was:

Do I need to train an AI model on the PDF?

No.

I built a system that lets an LLM answer questions about a document without fine-tuning the LLM on that document.

The basic idea is called Retrieval-Augmented Generation, or RAG.

And once I understood how it worked, the architecture was surprisingly straightforward.

What I Wanted to Build

The goal was simple:

Upload PDF
     ↓
Ask a question
     ↓
Find the relevant parts of the PDF
     ↓
Give those parts to the LLM
     ↓
Generate an answer

For example, imagine I upload a 100-page research paper.

I ask:

"What dataset did the authors use?"

I don't want the LLM to process the entire document from scratch every time.

Instead, my system should find the section containing the dataset information and send only that relevant context to the LLM.

That is the basic idea behind RAG.

The Architecture

The complete pipeline looks like this:

                  PDF
                   │
                   ▼
            Text Extraction
                   │
                   ▼
             Text Chunking
                   │
                   ▼
          Embedding Generation
                   │
                   ▼
            Vector Database
                   │
                   │
        ┌──────────┘
        │
        ▼
   User Question
        │
        ▼
 Question Embedding
        │
        ▼
 Similarity Search
        │
        ▼
 Relevant Chunks
        │
        ▼
      LLM
        │
        ▼
     Answer

There are two important phases here.

Phase 1: Indexing

The PDF is processed and stored in a searchable format.

Phase 2: Retrieval + Generation

When the user asks a question, the system retrieves the relevant information and gives it to the LLM.

Let's break that down.

Step 1: Extract Text From the PDF

The first thing we need is the actual text.

For a normal text-based PDF, a library such as PyMuPDF can extract it.

A simplified example:

import fitz

def extract_text(pdf_path):
    document = fitz.open(pdf_path)

    pages = []

    for page in document:
        pages.append(page.get_text())

    return "\n".join(pages)

Now we have something like:

Introduction...

Related Work...

Methodology...

Dataset...

Experiments...

Results...

Conclusion...

But we aren't ready to send this entire text to the LLM.

There could be thousands or hundreds of thousands of words.

So the next step is important.

Step 2: Split the Document Into Chunks

Instead of treating the entire PDF as one giant piece of text, we divide it into smaller chunks.

For example:

PDF
│
├── Chunk 1
├── Chunk 2
├── Chunk 3
├── Chunk 4
├── Chunk 5
└── ...

Why?

Because when someone asks a question, we don't necessarily need the entire document.

Suppose the PDF contains this:

Page 1
Introduction...

Page 2
Related Work...

Page 3
Dataset...

Page 4
Methodology...

Page 5
Training...

Page 6
Results...

If the user asks:

"What dataset was used?"

We only need the relevant portion.

A simple chunking strategy could look like:

def create_chunks(text, chunk_size=1000, overlap=200):
    chunks = []

    start = 0

    while start < len(text):
        end = start + chunk_size
        chunks.append(text[start:end])
        start += chunk_size - overlap

    return chunks

The overlap is useful because an important sentence might otherwise fall exactly between two chunks.

For a real project, chunk size should be tested rather than blindly chosen.

Step 3: Convert Text Into Embeddings

Now comes one of the most important parts.

A computer cannot directly perform semantic similarity search on ordinary sentences.

We convert each chunk into a numerical representation called an embedding.

For example:

"What dataset was used?"

might become something conceptually like:

[0.21, -0.18, 0.74, 0.03, ...]

The actual embedding contains many dimensions.

The important idea is that semantically similar text should have similar vector representations.

For example:

Question:
"What dataset did the researchers use?"

Document chunk:
"The experiments were conducted using the WESAD dataset..."

These two pieces of text are semantically related even though they don't use exactly the same words.

That's why embeddings are useful.

Step 4: Store the Embeddings

Now we need somewhere to store the vectors.

A vector database or vector index can be used for this.

For a small local project, FAISS is one option.

Conceptually:

Chunk 1 → Embedding 1
Chunk 2 → Embedding 2
Chunk 3 → Embedding 3
Chunk 4 → Embedding 4
...

We store the relationship between the vector and its original text.

So later, when we find a relevant vector, we can retrieve the original chunk.

The system is essentially building a searchable representation of the PDF.

Step 5: The User Asks a Question

Now the interesting part begins.

Suppose I ask:

"What dataset was used in the experiments?"

The question itself is converted into an embedding.

User Question
      ↓
Embedding Model
      ↓
Question Vector

We then compare that vector against the vectors stored from the PDF.

The goal is to find the chunks that are semantically closest to the question.

Step 6: Retrieve the Relevant Chunks

Suppose the search returns:

Chunk 18
Chunk 42
Chunk 43
Chunk 67

The system can select the top few results.

For example:

Question
   ↓
Vector Search
   ↓
Top 5 Relevant Chunks

We now have the context needed to answer the question.

This is the retrieval part of Retrieval-Augmented Generation.

Step 7: Give the Context to the LLM

Now we finally use the language model.

Instead of asking:

"What dataset was used?"

we provide the retrieved information as context.

Conceptually:

Use the following context to answer the question.

Context:
[Relevant chunk 1]

[Relevant chunk 2]

[Relevant chunk 3]

Question:
What dataset was used?

Answer:

The LLM can now generate an answer based on the retrieved document content.

This is where the "generation" part of RAG comes in.

The Complete Flow

Putting everything together:

                  ┌──────────────┐
                  │     PDF      │
                  └──────┬───────┘
                         │
                         ▼
                ┌─────────────────┐
                │ Text Extraction │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │    Chunking     │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │   Embeddings    │
                └────────┬────────┘
                         │
                         ▼
                ┌─────────────────┐
                │  Vector Index   │
                └─────────────────┘


User Question
      │
      ▼
┌─────────────────┐
│ Question        │
│ Embedding       │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Similarity      │
│ Search          │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│ Relevant        │
│ Document Chunks │
└────────┬────────┘
         │
         ▼
┌─────────────────┐
│      LLM        │
└────────┬────────┘
         │
         ▼
      Answer

That's the entire idea.

It sounds complicated when people say "build a RAG pipeline."

The individual steps are actually quite understandable.

Why Not Just Put the Entire PDF Into the LLM?

This is a reasonable question.

Modern LLMs can process large amounts of text.

So why bother with retrieval?

Because document-based applications have several practical problems.

A document may be:

Very large
Frequently updated
Made up of many unrelated sections
Too expensive to repeatedly process in full
Full of information irrelevant to the current question

Retrieval lets us narrow the information down before generation.

Instead of:

100-page document
      ↓
      LLM

we can do:

100-page document
      ↓
Find relevant information
      ↓
5 useful chunks
      ↓
LLM

The second approach gives the model a much more focused context.

The Most Important Problem: Hallucinations

At this point, the chatbot looks impressive.

But there is a problem.

What happens if the answer isn't actually in the PDF?

Suppose I ask:

"What did the authors say about a technology that isn't mentioned anywhere in the paper?"

A language model might still try to answer.

That's dangerous.

A document chatbot should not confidently invent information simply because the user asked a question.

So one of the most important rules I would add is:

If the retrieved context does not contain enough information to answer the question, say so.

For example:

I couldn't find enough information in the provided
document to answer this question.

That's much better than generating a convincing but unsupported answer.

I Would Test the Chatbot With Questions Like These

Instead of testing only easy questions, I would deliberately try to break it.

Test 1 — Direct question

"What dataset was used?"

Expected:

A direct answer based on the document.

Test 2 — Multiple sections

"How does the proposed method differ from the baseline?"

This may require retrieving information from more than one section.

Test 3 — Missing information

"What programming language was used to build the company's mobile application?"

If the PDF never discusses this, the system should say that it cannot find the answer.

Test 4 — Ambiguous question

"What was the result?"

The document may contain multiple results.

The chatbot should ideally ask for clarification or use the surrounding context.

Test 5 — Misleading question

"Why did the researchers use Dataset X?"

If Dataset X doesn't exist in the document, the system shouldn't accept the assumption as fact.

This type of testing is much more interesting than simply asking:

"What is the title of the paper?"

RAG vs Fine-Tuning

This was one of the biggest things I wanted to understand.

These two approaches solve different problems.

Fine-tuning

Fine-tuning changes the model's behavior by training it further on a dataset.

Conceptually:

Base Model
    ↓
Training Data
    ↓
Fine-Tuning
    ↓
Modified Model

RAG

RAG doesn't require the model to learn the document's contents during training.

Instead:

Document
    ↓
Index

Question
    ↓
Retrieve relevant information
    ↓
LLM

That makes RAG particularly useful for document collections that change frequently.

If I add another PDF, I don't necessarily need to retrain the language model.

I can process the new document and add its chunks to the retrieval system.

A Simple Project Structure

A project like this can be organized fairly cleanly:

pdf-chatbot/
│
├── app.py
├── ingest.py
├── retriever.py
├── generator.py
├── embeddings.py
│
├── data/
│   └── documents/
│
├── index/
│   └── vector_store/
│
├── requirements.txt
└── README.md

For example:

ingest.py

handles:

PDF
→ extraction
→ chunking
→ embeddings
→ indexing

While:

retriever.py

handles:

question
→ embedding
→ similarity search
→ relevant chunks

And:

generator.py

handles:

context + question
→ LLM
→ answer

Keeping those responsibilities separate makes the project easier to debug.

What I Would Put in the UI

The interface doesn't need to be complicated.

Something like:

┌─────────────────────────────────────────┐
│           PDF Question Answering        │
├─────────────────────────────────────────┤
│                                         │
│       [ Upload PDF ]                    │
│                                         │
│  ─────────────────────────────────────  │
│                                         │
│  Ask a question:                        │
│  ┌───────────────────────────────────┐  │
│  │ What dataset was used?            │  │
│  └───────────────────────────────────┘  │
│                                         │
│              [ Ask ]                    │
│                                         │
├─────────────────────────────────────────┤
│ Answer                                  │
│                                         │
│ The experiments used ...                │
│                                         │
├─────────────────────────────────────────┤
│ Sources                                 │
│                                         │
│ Page 4                                  │
│ Page 7                                  │
└─────────────────────────────────────────┘

The source section is particularly useful.

Instead of simply saying:

"Here is the answer."

the application can show:

"Here are the document sections used to generate this answer."

That makes the system easier to inspect.

What I Learned

The interesting thing about this project wasn't actually the chatbot.

It was understanding the separation between knowledge retrieval and language generation.

The LLM doesn't necessarily need to memorize every document.

It can be given the relevant information when the question arrives.

That changes the way you think about building AI applications.

Instead of asking:

"How do I train an AI to know everything in this document?"

you can ask:

"How do I efficiently retrieve the right information and give it to the model?"

That is a much more practical engineering problem.

Where This Can Be Used

The same architecture can be adapted to many applications.

Research papers

Upload papers and ask questions about:

methodology
datasets
experiments
limitations
results

College notes

Upload lecture notes and ask:

"Explain Unit 3 in simple terms."

Company documentation

Upload internal documentation and search it conversationally.

Legal documents

Retrieve relevant sections from large documents.

Product manuals

Ask:

"How do I reset this device?"

Technical documentation

Ask questions without manually searching through hundreds of pages.

The basic architecture remains similar.

What I Would Improve Next

The first version of a RAG system is relatively simple.

A production-quality version is much harder.

There are several things I would improve.

1. Better chunking

Fixed character lengths aren't always ideal.

A chunk should ideally preserve meaningful context.

2. Better retrieval

Basic similarity search isn't always enough.

Hybrid search can combine semantic similarity with keyword-based retrieval.

3. Re-ranking

Instead of immediately passing the top results to the LLM, a re-ranking model can help determine which retrieved chunks are actually the most relevant.

4. Source citations

The chatbot should tell the user exactly where its answer came from.

5. Better handling of tables

PDFs aren't just text.

Tables, figures, columns, and scanned pages can make extraction much harder.

6. Evaluation

A serious RAG application needs evaluation.

I'd want to measure things such as:

Retrieval accuracy
Answer correctness
Context relevance
Hallucination rate
Response latency

Without evaluation, "it seems to work" isn't enough.

The Bigger Picture

The interesting part about RAG isn't that it lets you "chat with PDFs."

That's just one application.

The larger idea is that an LLM can be connected to an external knowledge source without having that knowledge permanently embedded into its parameters.

The architecture becomes:

External Knowledge
       ↓
    Retrieval
       ↓
    Context
       ↓
      LLM
       ↓
    Response

And that pattern can be used far beyond PDFs.

That's what made this project interesting to me.

I started with a simple question:

"Can I make an AI answer questions about my PDF?"

The answer was yes.

But the more interesting realization was:

I didn't need to train the AI to know the document.

I just needed to build a good way for the AI to find the right information.

And that is the basic idea behind Retrieval-Augmented Generation.

Final Takeaway

If you're building your first document-based AI application, don't immediately think about fine-tuning.

Start by asking:

Can I retrieve the right information
and give it to the model at the right time?

If the answer is yes, you may already have most of what you need.

The model generates the language.

The retrieval system finds the knowledge.

And the application connects the two.

That's RAG.