neural net training
Training a neural network involves 3 steps: 1) Create the neural network (covered in previous note) 2) Determine the loss function/cost function to be used.
- If dealing with a binary classification problem, can use binary cross entropy
- If regression, can use mean squared error
3) Use gradient descent to update weights to minimize loss/cost function.
- When it comes to activation functions, these are the commonly used functions:
- ReLU
- The function is g(z) = max(0, z)
- Linear Activation Function.
- The function is g(z) = z
- Sigmoid Function
- The function is g(z) = 1/(1+e^(-z))
We need activation functions in a neural network different than just linear functions in order to capture non-linear patterns in the data.
There's a reason the field has progressed towards using ReLU activation functions (especially within the hidden layers). It's because: 1) Compared to sigmoid function which includes exponentials, it is quicker to compute ReLU and this leads to more efficiency during gradient descent. 2) The sigmoid function has two parts of the graph that are flat, so minimizing the loss with gradient descent process is harder than in ReLU which only has a single flat portion in the curve. This causes a vanishing gradient problem because the partial derivative in these regions would be close to 0 in these flat regions which means that weight updates would stall. However, for the output layer (depending on the final answer), the above activation functions may be useful in those cases.