Building a neural network without a framework is a useful way to understand forward propagation, backpropagation, activation functions, and weight updates as one working system. This article implements a small fully connected network in C# and trains it to solve XOR and XNOR.
Prerequisites and Scope
This is a historical article from 2017, not production ML guidance. It targets C# 7 and .NET Framework 4.7, deliberately omits bias terms, and preserves the original code—including its comments and formatting. Treat it as an under-the-hood learning exercise; for a production system, use a maintained ML framework and current hardware support.
The walkthrough assumes that you already understand the basic mathematics of neural networks. If you need a deeper foundation, Simon Haykin's Neural Networks: A Comprehensive Foundation explains the mechanics in detail.
I originally built a handwritten-digit recognizer as a school project, then reduced the idea to a network small enough to inspect piece by piece. The smaller problem makes every part of the training loop visible instead of hiding it behind a framework API.
Neural Network Architecture
Before writing any code, you should draw the network on paper. That makes its structure and behavior easier to visualize. My sketch became the diagram below. And yes, this is a console application in Visual Studio 2017 targeting .NET Framework 4.7.
Network at a glance
- Multilayer fully connected perceptron.
- One hidden layer.
- Four neurons in the hidden layer—the perceptron converged with this number.
- Training algorithm: backpropagation.
- Stopping criterion: the mean squared error for an epoch falls below a threshold of 0.001.
- Learning rate: 0.1.
- Activation function: logistic sigmoid.
Next, we need somewhere to store the weights, perform calculations, do a little debugging, and make some use of tuples. These are our using directives.
Weight Storage Files
The project's release or debug directory contains one file per layer, named something like (fieldname)_memory.xml. You can probably guess what they are for. The files are created in advance with the total number of weights in each layer. I know XML is not the best parsing format; I simply did not have much time.
using System.Xml;
using static System.Math;
using static System.Console;
Implementing the Neural Network in C
We have two kinds of computational neurons: hidden and output. Weights can be read from or written to storage. We represent those concepts with two enums.
enum MemoryMode
{
GET,
SET
}
enum NeuronType
{
Hidden,
Output
}
Everything else lives in a namespace I will simply call NeuralNetwork.
Input Layer and XOR/XNOR Training Data
First, why did I draw the input-layer neurons as squares? They calculate nothing. They capture information from the outside world—the signal that will pass through the network—so the input layer has little in common with the others.
Should it get a separate class? For image, video, or audio processing, it should: the class gives you somewhere to transform and normalize data into the form expected by the network. That is why I will write an InputLayer class after all. It contains a training set organized in an unusual structure. The first array in each tuple contains combinations of 1 and 0. The second contains the corresponding XOR and XNOR results, in that order.
class InputLayer
{
private (double[], double[])[] _trainset = new (double[], double[])[]
{
(new double[] { 0, 0 }, new double[] { 0, 1 }),
(new double[] { 0, 1 }, new double[] { 1, 0 }),
(new double[] { 1, 0 }, new double[] { 1, 0 }),
(new double[] { 1, 1 }, new double[] { 0, 1 }),
};
public (double[], double[])[] Trainset
{
get => _trainset;
}
}
Neurons, Activation, and Gradients
Now for the most important part, without which no neural network can become a Terminator: the neuron. I will omit bias terms here. The neuron resembles the McCulloch-Pitts model, but replaces the threshold function with another activation function. It also has methods for calculating gradients and derivatives, its own type, and combined linear and nonlinear transforms. Naturally, we also need a constructor.
class Neuron
{
public Neuron(double[] inputs, double[] weights, NeuronType type)
{
_type = type;
_weights = weights;
_inputs = inputs;
}
private NeuronType _type;
private double[] _weights;
private double[] _inputs;
public double[] Weights
{
get => _weights;
set => _weights = value;
}
public double[] Inputs
{
get => _inputs;
set => _inputs = value;
}
public double Output
{
get => Activator(_inputs, _weights);
}
private double Activator(double[] i, double[] w)
{
double sum = 0;
for (int l = 0; l < i.Length; ++l)
sum += i[l] * w[l];
return Pow(1 + Exp(-sum), -1);
}
public double Derivativator(double outsignal) => outsignal * (1 - outsignal);
public double Gradientor(double error, double dif, double g_sum) =>
(_type == NeuronType.Output) ? error * dif : g_sum * dif;
}
Hidden and Output Layers
We have neurons, but they need to be grouped into layers for computation. Look back at the diagram and note the black dashed line. It separates the layers to show what each contains. A computational layer contains neurons and the weights connecting them to the previous layer's neurons.
Neurons are stored in an array rather than a list to reduce overhead. The weights form a matrix—a two-dimensional array—with dimensions [number of neurons in the current layer **x** number of neurons in the previous layer]. The layer must initialize its neurons or we will get a null reference. The layers are structurally similar but differ in their logic, so the hidden and output layers should inherit from one abstract base class.
abstract class Layer
{
protected Layer(int non, int nopn, NeuronType nt, string type)
{
numofneurons = non;
numofprevneurons = nopn;
Neurons = new Neuron[non];
double[,] Weights = WeightInitialize(MemoryMode.GET, type);
for (int i = 0; i < non; ++i)
{
double[] temp_weights = new double[nopn];
for (int j = 0; j < nopn; ++j)
temp_weights[j] = Weights[i, j];
Neurons[i] = new Neuron(null, temp_weights, nt);
}
}
protected int numofneurons;
protected int numofprevneurons;
protected const double learningrate = 0.1d;
Neuron[] _neurons;
public Neuron[] Neurons
{
get => _neurons;
set => _neurons = value;
}
public double[] Data
{
set
{
for (int i = 0; i < Neurons.Length; ++i)
Neurons[i].Inputs = value;
}
}
public double[,] WeightInitialize(MemoryMode mm, string type)
{
double[,] _weights = new double[numofneurons, numofprevneurons];
WriteLine($"{type} weights are being initialized...");
XmlDocument memory_doc = new XmlDocument();
memory_doc.Load($"{type}_memory.xml");
XmlElement memory_el = memory_doc.DocumentElement;
switch (mm)
{
case MemoryMode.GET:
for (int l = 0; l < _weights.GetLength(0); ++l)
for (int k = 0; k < _weights.GetLength(1); ++k)
_weights[l, k] = double.Parse(
memory_el
.ChildNodes.Item(k + _weights.GetLength(1) * l)
.InnerText.Replace(',', '.'),
System.Globalization.CultureInfo.InvariantCulture
);
break;
case MemoryMode.SET:
for (int l = 0; l < Neurons.Length; ++l)
for (int k = 0; k < numofprevneurons; ++k)
memory_el.ChildNodes.Item(k + numofprevneurons * l).InnerText = Neurons[l]
.Weights[k]
.ToString();
break;
}
memory_doc.Save($"{type}_memory.xml");
WriteLine($"{type} weights have been initialized...");
return _weights;
}
public abstract void Recognize(Network net, Layer nextLayer);
public abstract double[] BackwardPass(double[] stuff);
}
Why abstract classes matter
Layer is abstract, so it cannot be instantiated. We preserve the properties of a layer through inheritance: the derived constructor calls the parent constructor with base and otherwise fits on one line, because all constructor logic is already defined in the base class and need not be repeated.
Now for the derived classes themselves, HiddenLayer and OutputLayer, presented together in one block.
class HiddenLayer : Layer
{
public HiddenLayer(int non, int nopn, NeuronType nt, string type)
: base(non, nopn, nt, type) { }
public override void Recognize(Network net, Layer nextLayer)
{
double[] hidden_out = new double[Neurons.Length];
for (int i = 0; i < Neurons.Length; ++i)
hidden_out[i] = Neurons[i].Output;
nextLayer.Data = hidden_out;
}
public override double[] BackwardPass(double[] gr_sums)
{
double[] gr_sum = null;
for (int i = 0; i < numofneurons; ++i)
for (int n = 0; n < numofprevneurons; ++n)
Neurons[i].Weights[n] +=
learningrate
* Neurons[i].Inputs[n]
* Neurons[i].Gradientor(0, Neurons[i].Derivativator(Neurons[i].Output), gr_sums[i]);
return gr_sum;
}
}
class OutputLayer : Layer
{
public OutputLayer(int non, int nopn, NeuronType nt, string type)
: base(non, nopn, nt, type) { }
public override void Recognize(Network net, Layer nextLayer)
{
for (int i = 0; i < Neurons.Length; ++i)
net.fact[i] = Neurons[i].Output;
}
public override double[] BackwardPass(double[] errors)
{
double[] gr_sum = new double[numofprevneurons];
for (int j = 0; j < gr_sum.Length; ++j)
{
double sum = 0;
for (int k = 0; k < Neurons.Length; ++k)
sum +=
Neurons[k].Weights[j]
* Neurons[k]
.Gradientor(errors[k], Neurons[k].Derivativator(Neurons[k].Output), 0);
gr_sum[j] = sum;
}
for (int i = 0; i < numofneurons; ++i)
for (int n = 0; n < numofprevneurons; ++n)
Neurons[i].Weights[n] +=
learningrate
* Neurons[i].Inputs[n]
* Neurons[i].Gradientor(errors[i], Neurons[i].Derivativator(Neurons[i].Output), 0);
return gr_sum;
}
}
Training with Backpropagation
The comments describe the important details. We now have every component: training and test data, computational elements, and layers. It is time to connect them through training. The algorithm is backpropagation, and the stopping criterion is a mean squared error below 0.001 for an epoch. The Network class holds the network state passed among the methods.
class Network
{
InputLayer input_layer = new InputLayer();
public HiddenLayer hidden_layer = new HiddenLayer(
4,
2,
NeuronType.Hidden,
nameof(hidden_layer)
);
public OutputLayer output_layer = new OutputLayer(
2,
4,
NeuronType.Output,
nameof(output_layer)
);
public double[] fact = new double[2];
double GetMSE(double[] errors)
{
double sum = 0;
for (int i = 0; i < errors.Length; ++i)
sum += Pow(errors[i], 2);
return 0.5d * sum;
}
double GetCost(double[] mses)
{
double sum = 0;
for (int i = 0; i < mses.Length; ++i)
sum += mses[i];
return (sum / mses.Length);
}
static void Train(Network net)
{
const double threshold = 0.001d;
double[] temp_mses = new double[4];
double temp_cost = 0;
do
{
for (int i = 0; i < net.input_layer.Trainset.Length; ++i)
{
net.hidden_layer.Data = net.input_layer.Trainset[i].Item1;
net.hidden_layer.Recognize(null, net.output_layer);
net.output_layer.Recognize(net, null);
double[] errors = new double[net.input_layer.Trainset[i].Item2.Length];
for (int x = 0; x < errors.Length; ++x)
errors[x] = net.input_layer.Trainset[i].Item2[x] - net.fact[x];
temp_mses[i] = net.GetMSE(errors);
double[] temp_gsums = net.output_layer.BackwardPass(errors);
net.hidden_layer.BackwardPass(temp_gsums);
}
temp_cost = net.GetCost(temp_mses);
WriteLine($"{temp_cost}");
} while (temp_cost > threshold);
net.hidden_layer.WeightInitialize(MemoryMode.SET, nameof(hidden_layer));
net.output_layer.WeightInitialize(MemoryMode.SET, nameof(output_layer));
}
static void Test(Network net)
{
for (int i = 0; i < net.input_layer.Trainset.Length; ++i)
{
net.hidden_layer.Data = net.input_layer.Trainset[i].Item1;
net.hidden_layer.Recognize(null, net.output_layer);
net.output_layer.Recognize(net, null);
for (int j = 0; j < net.fact.Length; ++j)
WriteLine($"{net.fact[j]}");
WriteLine();
}
}
static void Main(string[] args)
{
Network net = new Network();
Train(net);
Test(net);
ReadKey();
}
}
Training Result and Limitations
The training result:
After these brain-breaking straightforward manipulations, we have the foundation of a working neural network. To make it do something else, change the InputLayer class and choose suitable network parameters for the new task.
That is all. I will be happy to answer questions in the comments, but for now I have other things to do.
P.S. If you want to try the code, click here.
UPDATE 1 (October 22, 2020): Good grief, that was a long time ago. I hope I never write articles like this again. At the time, I probably wanted to share code like this with the community, but nobody writes ML this way.
UPDATE 2 (December 17, 2022): Recognizing 3 x 5 pixel images
Follow Stepami on GitHub for open-source C#/.NET projects, compiler experiments, production-focused examples, and new releases.
Top comments (0)