Igor's Website - Articles - Parallel Neural Network Library (.NET and C++)

Science, stories, art and music.

Science / Computer science articles.

Parallel Neural Network Library (.NET and C++)

Purpose

This article is intended to briefly explain the parallelization of an artificial neural network and present a freely available library for using artificial neural networks in .NET and C++ programs.

Introduction to neural networks

An artificial neural network is an artificial system designed for pattern recognition and classification. It is inspired by biological neural networks and has been used in many applications.

Essentially, an artificial neural network consists of interconnected artificial neurons. Neurons are usually grouped into several layers and all neurons from one layers are connected to the other. For example, given a three layered artificial neural network, the first layer could be connected to the second, and second to the third.

The impulses that are fed into the first layer are then propagated through network layers to the last layer (usually referred to as the output layer). The output depends on the weights of the connections between neurons. Now, the interesting part here is that the neural network can be trained to output specific values given a specific input.

A set of input patterns is used in the input layer of the network, and the network is then trained using error backpropagation (that is, the difference from the expected result is propagated back through the artificial neural network to correct the weights of the connections).

If you are a beginner, than this article may not be of much help to you. This article consernes mostly the parallelization of the artificial neural network and the implementation of a library.

Activation function

Artificial neurons of a neural network have an activation function which takes the outputs of the neurons from previous layer (since each neuron from any layer is connected to each neuron in adjacent layers) and calculates the output. This output is then propagated through the connections to the next layer and so on. The activation function can be a sigmoidal, tangent, linear or some other function, depending on the use. You can find more on activaton functions on the Wikipedia page about activation functions.

I will not go into these details as I assume you have some knowledge of these concepts. This article concerns parallelization of the neural network and the implementation of a library.

I will, however, point out some parts of the backpropagation algorithm, as this is the likely part of the neural networks to misunderstand. I have seen a lot of articles talk about the backpropagation, but the most confusing part was the adjustment of the weights during the backpropagation. There is some math involved, but if you need a quick explanation, you can observe the code fragments below.

Calculating deltas:
 float sum = 0;
 for (int r = 0; r < layers[i + 1].Length; ++r)
     sum += layers[i + 1][r].delta * connections[i][j, r];
 layers[i][j].delta = Derivative(layers[i][j].y) * sum;
 
Adjusting weights:
 for (int r = 0; r < connections[i].GetLength(1); ++r)
     connections[i][j, r] += learningRate * layers[i][j].y * layers[i + 1][r].delta;
 

Where connections[i] is a matrix of connections between the i-th and (i+1)-th layer of neurons, y is the last output and delta is the calculated error for a given neuron (where layers[i] is a vector of neurons in the i-th layer).

An important point here is that the Derivative(float) is the derivative of the activation function. Sigmoid functions have simple derivatives, which are elegant to use.

Connections and parallelization

Using matrices of connections between the layers allows for easy parallelization of the algorithm. Propagation of input (or error) between two layers can be safely parallelized for any matrix of connections. The important thing is to keep the order of layers, using parallelization only on the inner for loops, to allow the propagation (or backpropagation) to be done correctly.

Parallelization of weight adjustment:
for (int i = 0; i < connections.GetLength(0); ++i)
    Parallel.For(0, connections[i].GetLength(0), j =>
    {
        for (int r = 0; r < connections[i].GetLength(1); ++r)
            connections[i][j, r] += learningRate * layers[i][j].y * layers[i + 1][r].delta;
    });

NeuralNetworkModel Class

I have also included a NeuralNetworkModel class which can be used to connect more than one neural network together, allowing for recurrent neural networks to be used or network with memory to be created. This class is only included in the .NET library. The archive contains a test project which demonstrates the usage of the NeuralNetworkModel class.

The basic usage is the following. Parallel neural networks are assigned to the instantiated NeuralNetworkModel using the index braces:

NeuralNetworkLibrary.NeuralNetworkModel nnm = new NeuralNetworkModel();
nnm[0] = new ParallelNeuralNetwork(1, 2, 4);
nnm[1] = new ParallelNeuralNetwork(1, 2, 1);
nnm[2] = new ParallelNeuralNetwork(1, 3, 2);
nnm[3] = new ParallelNeuralNetwork(1, 2, 2);

Then, the input is mapped to neural networks:

nnm.AddInputVector(new int[] { 0, 0 });

Next, communication vectors are added. These vector map outputs from a single neural network to one or more others.

nnm.AddCommunicationVector(0, new int[] { 1, 1, 2, 2 });
nnm.AddCommunicationVector(1, new int[] { 3 });
nnm.AddCommunicationVector(2, new int[] { 3, 2 });
nnm.AddCommunicationVector(3, new int[] { -1, -1 });

The -1s indicates that the 3-rd neural network has both outputs mapped to the model's output. The last thing to add is the output mapping.

nnm.AddOutputVector(new NeuralNetworkModel.Relation[]
                   { 
                        new NeuralNetworkModel.Relation() { network = 3, index = 0 }, 
                        new NeuralNetworkModel.Relation() { network = 3, index = 1 } 
                   });

The relationships are one-to-one. The neural network model is then used like a neural network by invoking the Propagate and Backpropagate methods.

There is a Windows Forms project in the ZIP file which demonstrates the usage of the library. It demonstrates the usage of the NeuralNetworkModel class for recognizing handwritten digits on the NIST database. The database is included with the application.

If you find any bugs, you can mail me at igor@igorsevo.com.

Libraries

I have compiled two libraries for Windows, a .NET (CLR) and a C++ static library. I have not yet compiled the libraries for Mono or under *nix. Libraries are available in the Resources section, or using the following links:

License

This library is licensed under Attribution Assurance License.

Links

You can refer to some of the following articles (which other people wrote) which detail the implementation of a neural network. I found these articles to be very helpful.

Created: 27.1.2013, Modified; 15.2.2013

Google Code Prettify

I use Google Prettify to format the source code in my articles. If the code is displaying in one line, you can try opening the page in a different browser.

Request software design

If you wish to have a specific application designed, contact me at software@igorsevo.com. If you want to know more about what I do, check out my home page and Science page.

Support this site

Suggest an article

You can suggest an article or ask a question on the Questions page.

Social