Simple linear regression using PyTorch | Learn AI Together | Page 1

harsh vapor Jul 9, 2023, 1:06 AM

#

Can someone tell me why this linear regression fails?


import torch
import numpy as np
import matplotlib.pyplot as plt
 
X = torch.arange(4, 20, 0.1).view(-1, 1)
func = -1/2 * X - 6.5
Y = func + 0.4 * torch.randn(X.size())
 
# defining the function for forward pass for prediction
def forward(x):
    return w * x + b
 
# evaluating data points with Mean Square Error.
def criterion(y_pred, y):
    return torch.mean((y_pred - y) ** 2)
 
w = torch.tensor(-10.0, requires_grad=True)
b = torch.tensor(-20.0, requires_grad=True)
 
step_size = 0.1
loss_list = []
iter = 20
 
for i in range (iter):    
    # making predictions with forward pass
    Y_pred = forward(X)
    # calculating the loss between original and predicted data points
    loss = criterion(Y_pred, Y)
    # storing the calculated loss in a list
    loss_list.append(loss.item())
    # backward pass for computing the gradients of the loss w.r.t to learnable parameters
    loss.backward()
    # updateing the parameters after each iteration
    w.data = w.data - step_size * w.grad.data
    b.data = b.data - step_size * b.grad.data
    # zeroing gradients after each iteration
    w.grad.data.zero_()
    b.grad.data.zero_()
    # priting the values for understanding
    print('{}, \t{}, \t{}, \t{}'.format(i, loss.item(), w.item(), b.item()))
 
# Plotting the loss after each iteration
plt.plot(loss_list, 'r')
plt.tight_layout()
plt.grid('True', color='y')
plt.xlabel("Epochs/Iterations")
plt.ylabel("Loss")
plt.show()

I have just tried replacing the numbers in equation. Code taken from (https://machinelearningmastery.com/training-a-linear-regression-model-in-pytorch/)

sand musk Jul 9, 2023, 1:27 AM

#

Your init values and learning rate are very large

#

This results in gradient explosion and you quickly get nan results

harsh vapor Jul 9, 2023, 1:27 AM

#

I have played with the learning rate by the way, like setting it to 0.01 helped a bit

#

And increasing the number of iterations

#

But what about parameters size

sand musk Jul 9, 2023, 1:27 AM

#

I tweaked all your values and it works well now

#

harsh vapor Jul 9, 2023, 1:27 AM

#

sand musk This results in gradient explosion and you quickly get nan results

Yes that's happens

harsh vapor Jul 9, 2023, 1:28 AM

#

sand musk

May I know the values?

sand musk Jul 9, 2023, 1:28 AM

#

import torch
import numpy as np
import matplotlib.pyplot as plt
import os
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

X = torch.arange(4, 20, 0.1).view(-1, 1)
func = -1/2 * X - 6.5
Y = func + 0.4 * torch.randn(X.size())
 
# defining the function for forward pass for prediction
def forward(x):
    return w * x + b
 
# evaluating data points with Mean Square Error.
def criterion(y_pred, y):
    return torch.mean((y_pred - y) ** 2)
 
# w = torch.tensor(-10.0, requires_grad=True)
# b = torch.tensor(-20.0, requires_grad=True)

w = torch.tensor(-0.5, requires_grad=True)
b = torch.tensor(-1.0, requires_grad=True)

# step_size = 0.1
step_size = 0.0001
loss_list = []
# iter = 20
iter = 2000
 
for i in range (iter):    
    # making predictions with forward pass
    Y_pred = forward(X)
    # calculating the loss between original and predicted data points
    loss = criterion(Y_pred, Y)
    # storing the calculated loss in a list
    loss_list.append(loss.item())
    # backward pass for computing the gradients of the loss w.r.t to learnable parameters
    loss.backward()
    # updateing the parameters after each iteration
    w.data = w.data - step_size * w.grad.data
    b.data = b.data - step_size * b.grad.data
    # zeroing gradients after each iteration
    w.grad.data.zero_()
    b.grad.data.zero_()
    # priting the values for understanding
    print('{}, \t{}, \t{}, \t{}'.format(i, loss.item(), w.item(), b.item()))
 
# Plotting the loss after each iteration
plt.plot(loss_list, 'r')
plt.tight_layout()
plt.grid('True', color='y')
plt.xlabel("Epochs/Iterations")
plt.ylabel("Loss")
plt.show()

#

lmk if that works

#

<3

harsh vapor Jul 9, 2023, 1:31 AM

#

sand musk lmk if that works

Works well!

#

Can you tell me why did you downgrade parameters so much?

#

And what does this mean?
os.environ["KMP_DUPLICATE_LIB_OK"]="TRUE"

sand musk Jul 9, 2023, 1:36 AM

#

Ah yes ignore that, it's to fix a bug I have with my current conda environment - but you don't need it

#

Your function is -0.5 x - 6.5

#

You are trying to predict those values of w and b

#

so starting at a similar scale to them means your gradients aren't large

#

Someone more math inclined can probably explain much better 😭

#

If you cheat and init with closer to those values you'll see the loss is substantially lower:
w = torch.tensor(-0.5, requires_grad=True)
b = torch.tensor(-6.0, requires_grad=True)

#

harsh vapor Jul 9, 2023, 1:53 AM

#

But what if someone started with huge values?

#Simple linear regression using PyTorch