Papiha Joharapurkar

multiclass neural networks

Update: Have really enjoyed brushing up on my AI/ML fundamentals and through my daily-note learning series, reviewing notes and fundamentals that I had known like the back of my hand at one point in time.

  • As discussed in previous notes, the activation functions used can vary especially for hidden and output layers
  • why are activation functions needed?
  • if only linear activation functions used in all units, no different than a linear regression model - linear function of a linear function is just linear
  • for multiclass classification, can use softmax function
  • the function involves computing each output, exponentiating the outputs and then divide each by the sum of all exponentiated outputs of other neurons (itself included)
  • There is a unique feature about this activation function compared to other activation functions, which is that each output requires all element-wise exponentiated outputs to be computed before computing the final proportion to generate final activations
  • With application in Tensorflow, need to apply numerical stability (linear activation function and then loss with SparseCategoricalCrossEntropy, from_logits). Then need to apply tf.nn.softmax to convert back to probabilities.
  • In general, Adam (Adaptive Moment estimation) Algorithm can adjust the learning rate/alpha so in gradient descent, you can reach minimum faster
  • each weight, bias will get it own learning rate
  • if parameters are moving in same direction, increase learning rate
  • but if parameters are oscillating, decrease learning rate

What are convolution layers?

  • Each unit will only receive a region/limited window of previous layer's activations, not all
  • reasons?
  • faster computation
  • less training data/less prone to overfitting
  • CNN = when a neural network has many convolutional layers

All daily logs