Stumped on how to debug Neural Network | Learn AI Together | Page 1

azure crest Feb 28, 2024, 1:28 AM

#

Here's an output:

Epoch: 0, Loss: 1.2502108497948865
Epoch: 1000, Loss: 1.1696434861402938
Epoch: 2000, Loss: 1.1724973232384837
Epoch: 3000, Loss: 1.1777942899813245
Epoch: 4000, Loss: 1.1799072442138687
Epoch: 5000, Loss: 1.180972162149576
Epoch: 6000, Loss: 1.1816273309850438
Epoch: 7000, Loss: 1.1820796199555927
Epoch: 8000, Loss: 1.1822986414568912
Epoch: 9000, Loss: 1.1823103893963272
main.py:73: RuntimeWarning: overflow encountered in exp
  return 1.0 / (1 + np.exp(-x))
[[0.99960975]]

marsh python Feb 28, 2024, 2:51 AM

#

Your loss is not going down rather up

#

Adjust learning rate and make it lower

#

Your lr is 0.005 try intervals of 0.0001 and if training is slow then go upward but multiples of 3

#

Like 0.003 then 0.001 ... 0.003 etc

azure crest Feb 28, 2024, 7:16 PM

#

loss goes up still at 0.0001

#

even at 0.00001 it goes up

magic knoll Feb 28, 2024, 7:33 PM

#

Have you calculated maybe one or two or three gradients by hand to make sure backward is correct (like, for numeric stability)

#

See https://stackoverflow.com/questions/51976461/optimal-way-of-defining-a-numerically-stable-sigmoid-function-for-a-list-in-pyth

Stack Overflow

optimal way of defining a numerically stable sigmoid function for a...

For a scalar variable x, we know how to write down a numerically stable sigmoid function in python:
def sigmoid(x):
if x >= 0:
return 1. / ( 1. + np.exp(-x) )
else:
retur...

azure crest Feb 28, 2024, 9:00 PM

#

did you mean ensuring positive and negative values are accounted for? all the values are positive so this shouldn't be an issue

magic knoll Feb 28, 2024, 9:02 PM

#

no, i mean manually do the math and make sure your gradients are what you expect for a small tensor, like 3 examples, maybe reduce the network size to check this.

How do you know all the values in the hidden layer are stable?

#Stumped on how to debug Neural Network