Cum

backpropagation.md

@@ -0,0 +1,58 @@
+# Backpropagation
+
+WIP
+
+{ Dunno if this is completely correct, I'm learning this as I'm writing it. ~drummyfish }
+
+Consider the following neural network:
+
+```
+     w000     w100
+  x0------y0------z0
+    \  _//  \  _//  \
+     \/ /w010\/ /w110\
+   b0/\/   b1/\/      \_E
+     \/\     \/\      /
+     /\_\w001/\_\w101/
+    /   \\  /   \\  /
+  x1------y1------z1
+     w011     w111
+```
+
+It has an input layer (neurons *x0*, *x1*), a hidden layer (neurons *y0*, *y1* and a bias *b0*) and an output layer (neurons *z0*, *z1* and a bias *b1*). At the end there is a total error *E* computed from the networks's output against the desired output (training data).
+
+Each non-input neuron is a function: e.g. the neuron *z0* can be seen as a function *z0(x) = activation(w100 * y0(x) + w110 * y1(x) + b1)*. Let's say the *activation* function is the normally used [logistic function](logistic_function.md) *actiovation(x) = 1/(1 + e^x)*. If you don't know what the fuck is going on see [neural networks](neural_network.md) first.
+
+Let's say the total error is computed as the squared error: *E = squared_error(z0) + squared_error(z1) = 1/2 * (z0 - z0_desired)^2 + 1/2 * (z1 - z1_desired)^2*.
+
+What is our goal now? To find the **[partial derivative](partial_derivative.md) of the whole network's total error function** (at the current point defined by the weights and biases), or in other words the **gradient** at the current point. I.e. from the point of view of the total error (which is just a number output by this system), the network is a function of 10 variables (weights *w000*, *w001*, ... and the biases *b0* and *b1*), and we want to find a derivative of this function in respect to each of these variables (that's what a partial derivative is) at the current point (i.e. with current values of the weights and biases). This will, for each of these variables, tell us how much (at what rate and in which direction) the total error changes if we change that variable by certain amount. Why do we need to know this? So that we can do a [gradient descent](gradient_descent.md), i.e. this information is kind of a direction in which we want to move (change the weights and biases) towards lowering the total error (making the network compute results which are closer to the training data).
+
+Backpropagation work by going "backwards" from the output towards the input. So, let's start by computing the derivative against the weight *w100*.
+
+*derivative(E,w100) = derivative(squared_error(z0),w100) + derivative(squared_error(z0),w100) = derivative(squared_error(z0),w100) + 0*
+
+Notice that the second part of the sum (*derivative(squared_error(z1),w100)*) became 0 because when deriving in respect to *w100*, this expression is seen as a constant (as it doesn't depend on w100) and the derivative of a constant is 0. Now let's continue. We will now utilize the **chain rule** which is a rule of derivation that says:
+
+*derivative(f(g(x)),x) = derivative(derivative(f(g(x)),g(x))) * derivative(g(x),x)*
+
+In order to simplify the following equation let *T = w100 * y0(x) + w110 * y1(x) + b1*. Applying this rule to the above gives us (for demonstration with all intermediate steps):
+*derivative(E,w100) = derivative(squared_error(z0),w100) = derivative(squared_error(activation(w100 * y0(x) + w110 * y1(x) + b1)),w100) = derivative(squared_error(z0),z0) * derivative(activation(T),T) * derivative(w100 * y0(x) + w110 * y1(x) + b1,w110)*
+
+Now we can compute all the three parts of the sum:
+
+*derivative(squared_error(z0),z0) = derivative(1/2 * (z0 - z0_desired)^2,z0) = z0_desired - z0*
+
+*derivative(activation(T),T) = derivative(1/(1 + e^T),T) = T * (1 - T)*
+
+*derivative(w100 * y0(x) + w110 * y1(x) + b1,w100) = y0(x)*
+
+**Now we have computed the derivative against w100**, i.e. we have a formula with only variables whose values we know, so we can plug the values in and compute the derivative as a number *w100'*:
+
+*w100' = (derivative(1/2 * (z0 - z0_desired)^2,z0) = z0_desired - z0) * derivative(1/(1 + e^T),T) = T * (1 - T) * y0(x)*
+
+TO BE CONTINUED
+
+
+
+
+