The Simplest Neural Network That Runs on the 1900’s computer

When we think of creating or augmenting new AI systems, we tend to imagine that compute power is an undeniable requirement; however, in this article you will come to realize that simple programs that run even on the CPUs of old computers can host an intelligent program within them. Before the beans are spilled and the truth bombs are dropped, let us ask ourselves. How did we come to believe that AI software requires massive infrastructure containing data centers the size of a city?

Those that require huge compute resources are large language models simply referred to as LLMs. This requirement is no longer an issue if the first L in the LLM is stripped off. This is because the engine that powers the likes of Gemini goes through machine learning training through models. These models require feeding them massive datasets (the entire internet, in some cases) and performing quadrillions of calculations. This process can take weeks or months and is energy- and compute-intensive, even when using highly optimized hardware. Besides, Deep Learning models are built on vast neural networks that require performing matrix multiplications simultaneously. However, no such large datasets are required for a machine to perform operations and decision-making intelligently.

I gave Gemini a simple prompt to create a simple program in Vanilla JS just to illustrate neural networks, and it spat out all the code, and the training took a matter of minutes in a few epochs. Included in the code are concepts like back-propagation, activation function and feed-forward. It began by assigning Neuron and Network classes which represented a single neuron and the layers of the neural network and then it went on by doing its magic using ML algorithms.

The Activation Function used by Gemini was the old school Sigmoid function unlike contemporary deep learning algorithms which happen to use ReLU. Hereunder is a snippet from the code. The link to this awfully simple NeuralNet code is available here in latest version or from here for a 1999 machine compatible code.

   class Neuron {
    constructor(numInputs) {
        this.weights = [];
        for (let i = 0; i < numInputs; i++) {
            this.weights.push(Math.random() * 2 - 1); // Initialize weights randomly between -1 and 1
        }
        this.bias = Math.random() * 2 - 1;
    }

    // Sigmoid activation function
    activate(sum) {
        return 1 / (1 + Math.exp(-sum));
    }

    // Derivative of the sigmoid function, used in backpropagation
    derivative(output) {
        return output * (1 - output);
    }

    feedForward(inputs) {
        let sum = this.bias;
        for (let i = 0; i < this.weights.length; i++) {
            sum += inputs[i] * this.weights[i];
        }
        this.output = this.activate(sum);
        return this.output;
    }
}

export default Neuron;

class Network {
    constructor(inputNodes, hiddenNodes, outputNodes) {
        this.inputNodes = inputNodes;
        this.hiddenNodes = hiddenNodes;
        this.outputNodes = outputNodes;

        // Initialize layers
        this.hiddenLayer = [];
        for (let i = 0; i < hiddenNodes; i++) {
            this.hiddenLayer.push(new Neuron(inputNodes));
        }

        this.outputLayer = [];
        for (let i = 0; i < outputNodes; i++) {
            this.outputLayer.push(new Neuron(hiddenNodes));
        }
    }

    // Propagate data forward
    feedForward(inputArray) {
        let hiddenOutputs = this.hiddenLayer.map(neuron => neuron.feedForward(inputArray));
        let outputOutputs = this.outputLayer.map(neuron => neuron.feedForward(hiddenOutputs));
        return outputOutputs;
    }

    // Train the network using backpropagation
    backpropagate(inputArray, targetArray) {
        // 1. Forward pass to get outputs
        this.feedForward(inputArray);

        // 2. Calculate output layer errors and deltas
        const outputDeltas = this.outputLayer.map((neuron, i) => {
            const error = targetArray[i] - neuron.output;
            return error * neuron.derivative(neuron.output);
        });

        // 3. Calculate hidden layer errors and deltas
        const hiddenDeltas = this.hiddenLayer.map((neuron, i) => {
            let error = 0;
            for (let j = 0; j < this.outputLayer.length; j++) {
                error += this.outputLayer[j].weights[i] * outputDeltas[j];
            }
            return error * neuron.derivative(neuron.output);
        });

        // 4. Update output layer weights and biases
        this.outputLayer.forEach((neuron, i) => {
            for (let j = 0; j < neuron.weights.length; j++) {
                neuron.weights[j] += LEARNING_RATE * outputDeltas[i] * this.hiddenLayer[j].output;
            }
            neuron.bias += LEARNING_RATE * outputDeltas[i];
        });

        // 5. Update hidden layer weights and biases
        this.hiddenLayer.forEach((neuron, i) => {
            for (let j = 0; j < neuron.weights.length; j++) {
                neuron.weights[j] += LEARNING_RATE * hiddenDeltas[i] * inputArray[j];
            }
            neuron.bias += LEARNING_RATE * hiddenDeltas[i];
        });
    }
}

export default Network;