Neural Networks: Developing a Really Simple Neuron in TypeScript

Neural networks, at their core, are complex systems built from simple mathematical units called neurons (or perceptrons).

Today, we will implement a very simple neuron in TypeScript.

For context, a neural network is composed of one or more layers. Each is composed of one or more neurons. But we’ll save neural network layers for another blog post.

Building a Simple Neuron / Perceptron

The simplest form of a neuron is the perceptron. Let’s create a perceptron service:

export class PerceptronService {
  #weights: number[];

  constructor(
    private activationService: BinaryStepService,
    inputSize: number = 0,
  ) {
    this.#weights =
      inputSize > 0 ? Array(inputSize).fill(0).map(() => Math.random() * 2 - 1) : [];
  }
}

Inputs, Weights, Biases, and Random Initialization

• Inputs: A neuron takes one or more inputs. Each input will be evaluated against a corresponding, preconfigured weight.
• Weights: Weights determine how much influence each input has on the neuron's output. In a future post, we will train the network. Training is accomplished by tweaking (updating) the weights to improve the responses that the neural network generates.
• Random Initialization: We want to start in a state in which the model produces seemingly random responses. As it is trained, each weight may be updated to “teach” the model to provide higher quality responses. To produce this initial randomness, it is recommended to initialize the model with random weights — in this case with values between -1 and 1.
• Bias: Many neurons use a bias, instead of a threshold. For simplicity, we will skip biases today.

Activation Functions: Binary Step Activation

A neural network needs to determine if a neuron should fire, also known as activation.

There are many different types of activation functions. One of the simplest activation functions is the binary step function.

This activation function works by comparing the input value against a threshold. If the input is greater than (exceeds) the threshold number, the neuron is activated, which we will indicate by returning the number 1. Otherwise, it outputs 0 for “not activated.”

export class BinaryStepService {
  constructor(private threshold: number = 0) {}

  activate(input: number): number {
    return (input > this.threshold) ? 1 : 0;
  }
}

The threshold allows us to control the sensitivity of the neuron. A lower threshold makes the neuron more likely to activate, while a higher threshold makes it more selective.

Predictions — What the Neuron Does with Each Input and Weight

To evaluate if the neuron should fire, each input and weight is evaluated.

The standard neuron computation process is:

1. Dot Product Calculation: Apply a mathematical function to the weights and inputs — see Dot Product - The Core of the Computation below.

2. Activation: We take the value returned by the dot product calculation. If this value exceeds the threshold, the activation function returns 1 to indicate that the neuron is activated.
   

predict(inputFeatures: number[]): number {
  if (this.#weights.length === 0) {
    throw new Error('Weights not set or initialized');
  }

  if (inputFeatures.length !== this.#weights.length) {
    throw new Error(`Input features length (${inputFeatures.length}) does not match weights length (${this.#weights.length})`);
  }

  const dotProduct = this.dotProduct(inputFeatures, this.#weights); 
  return this.activationService.activate(dotProduct);
}

Dot Product — The Core of the Computation

Earlier, we assigned a weight to each input. Now, we multiply each weight by its corresponding input and sum up the results into a single number. This calculation is a formula from linear algebra called the dot product.

Every neuron will run this calculation. A neural network may customize this method or use an existing math package to optimize its performance.

private dotProduct(a: number[], b: number[]): number {
  if (a.length !== b.length) {
    throw new Error('Vectors must have the same length');
  }

  return a.reduce((sum, val, i) => sum + val * b[i], 0);
}

Weight Management

These getters and setters allow external training algorithms to get the neuron’s current weights, and update the weights with their learnings. The getter returns a copy of the weights, so that they are not accidentally modified.

setWeights(weights: number[]): void {
  this.#weights = weights;
}

getWeights(): number[] {
  return [...this.#weights];
}

Putting It All Together

Here is the full code with a couple of example usages for this simple neuron:

export class BinaryStepService {
  constructor(private threshold: number = 0) {}

  activate(input: number): number {
    return (input > this.threshold) ? 1 : 0;
  }
}

export class PerceptronService {
  #weights: number[];

  constructor(
    private activationService: BinaryStepService,
    inputSize: number = 0,
  ) {
    this.#weights =
      inputSize > 0 ? Array(inputSize).fill(0).map(() => Math.random() * 2 - 1) : [];
  }

  setWeights(weights: number[]): void {
    this.#weights = weights;
  }

  getWeights(): number[] {
    return [...this.#weights];
  }

  predict(inputFeatures: number[]): number {
    if (this.#weights.length === 0) {
      throw new Error('Weights not set or initialized');
    }

    if (inputFeatures.length !== this.#weights.length) {
      throw new Error(`Input features length (${inputFeatures.length}) does not match weights length (${this.#weights.length})`);
    }

    const dotProduct = this.dotProduct(inputFeatures, this.#weights); 

    console.log(dotProduct);
    return this.activationService.activate(dotProduct);
  }

  private dotProduct(a: number[], b: number[]): number {
    if (a.length !== b.length) {
      throw new Error('Vectors must have the same length');
    }

    return a.reduce((sum, val, i) => sum + val * b[i], 0);
  }
}

Example: Neuron Does Not Fire / Activate

// Do not activate the neuron unless the prediction exceeds 0.7
const activation = new BinaryStepService(0.7);

// Create a perceptron with 2 inputs and randomize the weights
const neuron = new PerceptronService(activation, 2);

// Set specific weights for each input
// This is typically done after we have started to train the model
// But let's set specific values so that we can see the math
neuron.setWeights([0.5, 0]);

// Make a prediction with input [0.8, 0.4]
const output = neuron.predict([0.8, 0.4]);
console.log(`Neuron output: ${output}`);

Produces:

Neuron output: 0 (is not activated)

Explanation:

• Input 1: 0.8 × Weight 1: 0.5 = 0.4
• Input 2: 0.4 × Weight 2: 0 = 0
• Sum = 0.4 + 0 = 0.4 < threshold 0.7 → no activation

Example: The Same Neuron Fires After its Weights have been Updated ("Training")

Let's say that training indicates the neuron should fire for these same inputs. Training updates the weights, and runs the calculation to see if the same inputs now activate the neuron. We will manually set the weights here, but a real neural network would use a method like backpropagation to update weights automatically.

Append this code to the script above:

// Update the weights to new values - increase the second weight from 0 to 0.9
neuron.setWeights([0.5, 0.9]);

// Make a prediction with the same input [0.8, 0.4]
const output2 = neuron.predict([0.8, 0.4]);
console.log(`Neuron output: ${output2}`);

Produces:

Neuron output: 1

Explanation:
0.8 × 0.5 = 0.4
0.4 × 0.9 = 0.36
Sum = 0.76 > threshold 0.7 → neuron fires

Why TypeScript — Shouldn’t You Have Used Python?

Yes, Python is the industry standard in artificial intelligence and neural network development. This post is just a quick prototype for TypeScript developers to get started with neural networks.

I would recommend a language like Python for its memory management, vast data science community, and scientific computing packages (NumPy, TensorFlow, PyTorch, etc.), but TypeScript is great for learning.