Neural Network & Genetic Algorithm

  1. Input is normalized in x;
  2. x is multiplied by a Weight(Dendrite);
  3. All x*weight are added by the Neuron in Sum;
  4. The Sum is put in a Sigmoid function and returns a Neuron Value;
  5. The Neuron Value is sent through another layer or the Output;

Input (Sensors)

Just as the name says, it gets the input from the world and sent it through the neural network. It’s however very important to normalize the values otherwise you couldn’t train the neural network. For that we could use a simple math function that is expressed like so:

Weights (Dendrites)

The line that connects one Neuron to another is the Weight. This is simply a number that the input is multiplied by. However it’s an important part, since the intelligence is all here. This weight number changes to learn and is carried through the iterations of training.

Initial weights

When the weights are first created, it’s important to make them as random as possible inside a range to avoid creating Nets that are too similar. I found out that a range between -4 and 4 is a good range to start.

Bound relaxation

Sometimes the Neural Net get stuck and can’t solve a problem past a certain point. I discovery that, if you increase your weights range from one generation to another, it improves the speed. So, after 100 generations (or iterations or epochs) you can increase the range from -4 and 4 to -5 and 5.

Hidden Layers (Neurons)

The Neurons are made of 2 important parts: Sum and Sigmoid. Each layers is made of an amount of neurons. The layers are usually 1 or 2 and there are very rare problems that can’t be solved in this manner.

Number of neurons

There is not a definitive rule as how many Neurons should we put on each layer. There are however some rules of thumb that can be used as required:

  • Should be between the size of the input layer and the size of the output layer;
  • Neurons (In+Out)/2;
  • Should be 2/3 the size of the input layer plus the size of the output layer;
  • Neurons = (In*0.6)+Out;
  • Should be less than twice the size of the input layer;
  • Neurons < In*2;

Sum

Once the Neuron receive the numbers from all the Dendrites, that is, after the input is multiplied by the weight, its is all added in a variable. This number is then sent to the sigmoid function.

Sigmoid

It’s the activation function.

  • Math: 1 / (1 + e-x);
  • Explain: 1 divided by the sum of 1 plus Euler(e) power by the negative input(x);
  • C#: return 1 / (1 + Math.Exp(-x));

Training

There are many algorithms like Back Propagation, Resilient, Newton’s method, Quasi Newton, Levenberg Marquardt, just to name a few. Most of training methods use pure math to calibrate the weights. The method that I present below, Genetic Algorithm, is a little different and use the sole power of many fast iterations to reach a satisfactory result.

Genetic Algorithm

This method simply converts the weights into “genes” and iterates the fittest populations until it reaches the desired result.

  1. Create a random population;
  2. Get the best nets accordingly to the fitness;
  3. Create the next generation based on the fittest;
  4. Introduce a small percentage of mutation in the new generation;
  5. Evaluate the best net on this generation;
  6. If the best fitness score is not good enough go back to Step 2;

Fitness

This calculates how far a given Neural Net is from a desired result (supervised training). That is, given some parameters(input) it has to reach an expected result (output).

Next Generation

The new neural nets will be generated based on the fittest of the past generation. It’s important to include these parents too in the new generation to avoid going backwards in fitness.

Mutation

Whenever you are geting the weights for the next generation, you should introduce a small chance to not get a weight from any parent and generate a random weight, a mutation!

Conclusion

Neural Networks have many diverse applications and it’s becoming ubiquitous on IT. There are many frameworks and tools that already implement them, but it’s important to know how it operates.

Example Solution

A workable version in C# of a Neural Network described on this article is in this link. I hope you enjoy!

Author

Cesar Ottani, programmer since 2001, Data Processing degree on FATEC São Paulo. Interested in game development and automation. Also a Rock drummer when possible.

Thanks to

My wife Vick for the dedication and patience.

--

--

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store