After tuning our model, we find that the softmax classifier works well. Specifically, the last layer of our network computes scores for each class, and they are fed into the softmax function. The model achieves 100% accuracy on the training data. However, we observe that the training loss does not reach zero. What can we say about (cross-entropy) loss can never be zero?
#Why can Cross-entropy not be zero?
gaunt rune
mental kayak
the only case that it goes to zero is the line (your NN) perfect fitting the observations, and is often an indication of overfitting. you can try to visualize that in a 2D graph using a simple linear model