Linear Regression using Pytorch from Scratch
We will start with Numpy and then look at Pytorch
import numpy as np
np.random.seed(42)
x = np.random.rand(100, 1)
# Below equation is y = a + bx + noise where a = 1 and b = 2
y = 1 + 2 * x + .1 * np.random.randn(100, 1)
# Shuffles the indices
idx = np.arange(100)
np.random.shuffle(idx)
# Uses first 80 random indices for train
train_idx = idx[:80]
# Uses the remaining indices for validation
val_idx = idx[80:]
# Generates train and validation sets
x_train, y_train = x[train_idx], y[train_idx]
x_val, y_val = x[val_idx], y[val_idx]
x_train[:5], y_train[:5]
print(type(x_train))
Above code cells are to create the data as Numpy arrays. This is the same data that we created in LR model with Numpy. https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy
We will convert the numpy arrays into Tensors in below cells so that we can use them with Pytorch.
!pip install torchviz
import torch
import torch.optim as optim
import torch.nn as nn
from torchviz import make_dot
#Setting the Device to CPU or GPU based on the availability
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print(device) #Since we are using GPU here, devide shows as cuda. Otherwise it would show as cpu.
# For using with Pytorch, we need to have PyTorch's Tensors
#
x_train_tensor = torch.from_numpy(x_train).float().to(device)
y_train_tensor = torch.from_numpy(y_train).float().to(device)
print(type(x_train), type(x_train_tensor), x_train_tensor.type())
We can see x_train_tensor is now a Tensor and .type() shows that it is a Tensor which is on GPU and is of Float datatype
# This is similar to what we did in Numpy.
# We set REQUIRES_GRAD = TRUE because we want to calculate partial derivatives or gradients on these parameters
a = torch.randn(1, requires_grad=True, dtype=torch.float)
b = torch.randn(1, requires_grad=True, dtype=torch.float)
print(a, b)
This looks good. These are Tensors and can be used with Pytorch. Is it ??
a.type()
This is a Float Tensor but resides on CPU. To use GPU, we need to push it to GPU.
a = a.to(device)
b = b.to(device)
a.type()
b.type()
Okay. Now we have pushed them to GPU.
a, b
What happened to requires_grad=True !? Looks like we lost it when we moved to GPU. So, this is not the right approach to create Tensors on GPU which require gradient descent
a = torch.randn(1, dtype=torch.float).to(device)
b = torch.randn(1, dtype=torch.float).to(device)
a.requires_grad_()
b.requires_grad_()
a, b
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
Okay.. Now we have a and b Tensors on GPU.. What next ? Start Forward Pass -> calculate Loss -> calculate gradients -> do backward pass and repeat it for number of epochs ??
lr = 1e-1
n_epochs = 1000
# set a learning rate and number of epochs
In this next section we will run a for loop and calculate the predictions, error and MSE loss similar to what we did before in LR with Numpy exercise.
However we will not calculate the gradient by hand. Rather we will use Pytorch's autograd function and its capabilities. Autograd is PyTorch’s automatic differentiation package and it will take care of calculating Partial Derivatives, Chain rule of derivatives where we multiply different partial derivatives to get a final derivative and so on.
It will also keep accumulating the gradient for each change of that object. We have set a and b to requires_grad=True. Therefore gradients will be calculated for them by Pytorch.
for epoch in range(n_epochs):
yhat = a + b * x_train_tensor
error = y_train_tensor - yhat
loss = (error ** 2).mean()
# Lets print a and b before the backward pass.
print("epoch number =", epoch)
print("a and b before backward pass" , a,b)
# We will not do this now like we did in Numpy notebook
# - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb
# a_grad = -2 * error.mean()
# b_grad = -2 * (x_tensor * error).mean()
# We will tell PyTorch to do the back progagation to the beginning from the specified loss which is at the end.
loss.backward()
# Let's check the computed gradients...
type(a.grad)
print("a's gradient = " , a.grad)
print("b's gradient = " , b.grad)
# Let's try to update the parameters with the formula we know - parameters = parameters - learning rate* gradient(parameters wrto Loss function)
a = a - lr * a.grad
b = b - lr * b.grad
print("a and b after backward pass" , a,b)
Well.. It did not work. a.grad and b.grad became None after epoch 0 and the parameter update statement fails for epoch 1. What happened ?
epoch number = 0 a and b before backward pass tensor([0.1940], device='cuda:0', requires_grad=True) tensor([0.1391], device='cuda:0', requires_grad=True)
See what happened after backward pass during epoch 0. The requires_grad=True vanished again after a and b were reassigned new values. This is similar to what happened before when we moved a and b from CPU to GPU. See here. Link to the cell - https://colab.research.google.com/drive/1OBbzaMpf33M8Fc-Z1yY0jqFKLScvYgco#scrollTo=mp9lvYynHw3U
a and b after backward pass tensor([0.5328], device='cuda:0', grad_fn= We used the inplace value substitution instead of assigning to a and b. There are two ways to do it and both are shown above i.e. a-= or b.sub_ It does not even finish epoch 0. It fails when assigning the value. This is because of Pytorch's Dynamic Computational Graph
Basically since required_grad is set to True for a and b, pytorch is calculating gradients and storing them for each change to a and b. Since we are trying to change the same object using its own value, it throws an error.
We need to basically somehow tell Pytorch not to calculate gradient for a and b when we are doing this parameter update step. This ran successfully. But the values of a and b are wrong at 900th epoch and is getting worse.
Our results from numpy exercise were very close to actual values after 1000 epochs.
This is another issue due to Pytorch's computational graph. It keeps accumulating the gradient values.
We need to tell it to let it go after an epoch. This looks good. Compare the gradient values for each epoch for this loop vs the one before.
The gradients keep improving here and the values of a and b keep coming closer to actual values. The predicted values are close to actual values. Now a and b are not messed up. But I am resetting them to check the Dynamic Computational Graph Lets manually calculate the prediction, error and loss for epoch 0 Blue box - This is for the tensors that are used as parameters i.e. a and b and we have set requires_grad = True for them. Grey box - This indicates some operation that involves a gradient-computing (like a or b) tensor and/or its dependencies. Green box - This indicates the starting point of the gradient calculation for the backward pass. This and grey boxes are the places where gradients are calculated. Note that there are no special mentions for x in the forward pass of the graph because it is not a tensor for which we compute gradients. Generally we do not update the weights/parameters manually since there will be a large number of them. We have to use the optim class of Pytorch. In below code, we are basically using the optim code to update parameters through SGD (in this case since entire batch of data is used it works as Batch Gradient Descent). In real life, this would actually work as a Mini Batch Gradient Descent since mini batches of data are sent at a time in each epoch. The output looks good and we used Pytorch's optim to update parameters and also reset the gradient after each epoch. Good going..! Lets work on Loss and Error first. Pytorch has many Loss functions and we have to use the appropriate ones based on the need. In this example, we are using MSE and we can use nn.MSELoss class for creating the loss function. For the model, we need to use the Module Class of Pytorch. We should create a new class which inherits from the Module class. It needs to have two mandatory methods apart from any other methods as required for the problem at hand - forward(self, x) - The method which does the actual prediction logic. Pytorch Module class is defined such that the forward method works as the core method of the class. When the class is called like a function, the forward method is invoked. Output same as before. We have 5 statements in the loop. Can we refactor these ? Lets refactor the training steps. Here, we just put all the code that kept repeating in the epoch loop in a method In real life, data will not be so simple. Therefore we pass data as (features, labels) to Pytorch features are inputs and labels are outputs. For this we use the Dataset class of Pytorch. Below we see a custom Dataset class that we defined by inheriting the Pytorch's Dataset class as well as TensorDataset class of Pytorch Any dataset returns three things, (x,y) , a way to index through x,y and length(x) The outputs of our Custom Dataset class and TensorDataset class are the same.
This is because we are doing exactly what the TensorDataset will do although our class will not have other bells and whistles which TensorDataset class will have. Okay.. What next ? In the above cells, we created Train and Valid Datasets using Pytorch TensorDataset class Also note that these Datasets are on cpu and not the GPU. We will get to know the reason for this once we understand Dataloaders Before that lets try to push the dataset to GPU.. Ahh... It fails.. Lets see what these Dataloaders are. Generally data is sent to GPU and to the training loop in mini batches. Dataloader is a class which can store the dataset and allows to fetch mini batches of data via iteration. Below gives a mini batch of data from train_data dataset. Above cell show the size of one iteration of the Dataloader. It is 16 i.e the size we specified. Above cells show the value of the index 0 in dataset from within the Loader as well as direclty from the dataset Well, we added an extra loop and it took slightly higher time to finish than our previous trials. However, this is required in real life, because the data will be huge and cannot be loaded into GPU at once. If you remember, we split the data 80:20 as train and valid in the beginning of this notebook. However, we generally use Pytorch's feature to split the data into train and valid (random split) or we have data split as training and validation data using some functional rule beforehand. We have trained the model, but we need to validate it. For that we use the validation dataloader-> dataset-> data How did this number come ?
valid data length (see first few lines of notebook) which is 20 / 16 batch size = 2 (approx) 2 * number of epochs(1000) = 2000 5 * number of epochs(1000) = 5000 As you can see the final losses of train and valid data, training loss is slightly higher than validation loss. OrderedDict([('a', tensor([1.0290], device='cuda:0')), ('b', tensor([1.9720], device='cuda:0'))])print(a, b)
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
print(a, b)
for epoch in range(n_epochs):
yhat = a + b * x_train_tensor
error = y_train_tensor - yhat
loss = (error ** 2).mean()
# Lets print a and b before the backward pass.
print("epoch number =", epoch)
print("a and b before backward pass" , a,b)
# We will not do this now like we did in Numpy notebook
# - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb
# a_grad = -2 * error.mean()
# b_grad = -2 * (x_tensor * error).mean()
# We will tell PyTorch to do the back progagation to the beginning from the specified loss which is at the end.
loss.backward()
# Let's check the computed gradients...
type(a.grad)
print("a's gradient = " , a.grad)
print("b's gradient = " , b.grad)
# Let's try to update the parameters with the formula we know - parameters = parameters - learning rate* gradient(parameters wrto Loss function)
a-= lr * a.grad
b.sub_(lr * b.grad)
print("a and b after backward pass" , a,b)
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
print(a, b)
for epoch in range(n_epochs):
yhat = a + b * x_train_tensor
error = y_train_tensor - yhat
loss = (error ** 2).mean()
# Lets print a and b before the backward pass for certain epochs only.
if(epoch%100==0):
print("epoch number =", epoch)
print("a and b before backward pass" , a,b)
# We will not do this now like we did in Numpy notebook
# - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb
# a_grad = -2 * error.mean()
# b_grad = -2 * (x_tensor * error).mean()
# We will tell PyTorch to do the back progagation to the beginning from the specified loss which is at the end.
loss.backward()
# Let's check the computed gradients...
# Lets print the gradients for certain epochs only.
if(epoch%100==0):
print("a's gradient = " , a.grad)
print("b's gradient = " , b.grad)
# Let's try to update the parameters with the formula we know - parameters = parameters - learning rate* gradient(parameters wrto Loss function)
# We are telling Pytorch not to calculate gradients for steps within the "with"
# torch.no_grad() allows us to perform regular Python operations on tensors, independent of PyTorch’s Dynamic computation graph.
with torch.no_grad():
a-= lr * a.grad
b.sub_(lr * b.grad)
# Lets print a and b after the backward pass for certain epochs only.
if(epoch%100==0):
print("a and b after backward pass" , a,b)
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
print(a, b)
for epoch in range(n_epochs):
yhat = a + b * x_train_tensor
error = y_train_tensor - yhat
loss = (error ** 2).mean()
# Lets print a and b before the backward pass for certain epochs only.
if(epoch%100==0):
print("epoch number =", epoch)
print("a and b before backward pass" , a,b)
# We will not do this now like we did in Numpy notebook
# - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb
# a_grad = -2 * error.mean()
# b_grad = -2 * (x_tensor * error).mean()
# We will tell PyTorch to do the back progagation to the beginning from the specified loss which is at the end.
loss.backward()
# Let's check the computed gradients...
# Lets print the gradients for certain epochs only.
if(epoch%100==0):
print("a's gradient = " , a.grad)
print("b's gradient = " , b.grad)
# Let's try to update the parameters with the formula we know - parameters = parameters - learning rate* gradient(parameters wrto Loss function)
# We are telling Pytorch not to calculate gradients for steps within the "with"
with torch.no_grad():
a-= lr * a.grad
b.sub_(lr * b.grad)
# Lets print a and b after the backward pass for certain epochs only.
if(epoch%100==0):
print("a and b after backward pass" , a,b)
# We reset it to zero after each epoch
a.grad.zero_()
b.grad.zero_()
# Lets print a and b after the backward pass and after reset to zero for certain epochs only.
if(epoch%100==0):
print("a and b after backward pass after resetting grad to zero" , a,b)
a, b
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
print(a, b)
yhat = a + b * x_train_tensor
error = y_train_tensor - yhat
loss = (error ** 2).mean()
make_dot(yhat)
make_dot(error)
make_dot(loss)
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
lr = 1e-1
n_epochs = 1000
# Define a SGD optimizer to update the parameters
optimizer = optim.SGD([a, b], lr=lr)
for epoch in range(n_epochs):
yhat = a + b * x_train_tensor
error = y_train_tensor - yhat
loss = (error ** 2).mean()
# Lets print a and b before the backward pass for certain epochs only.
if(epoch%100==0):
print("epoch number =", epoch)
print("a and b before backward pass" , a,b)
# We will not do this now like we did in Numpy notebook
# - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb
# a_grad = -2 * error.mean()
# b_grad = -2 * (x_tensor * error).mean()
# We will tell PyTorch to do the back progagation to the beginning from the specified loss which is at the end.
loss.backward()
# Let's check the computed gradients...
# Lets print the gradients for certain epochs only.
if(epoch%100==0):
print("a's gradient = " , a.grad)
print("b's gradient = " , b.grad)
# Let's try to update the parameters with the formula we know - parameters = parameters - learning rate* gradient(parameters wrto Loss function)
# We are telling Pytorch not to calculate gradients for steps within the "with"
#with torch.no_grad():
# a-= lr * a.grad
# b.sub_(lr * b.grad)
# We will use optimizer to update the parameters.
optimizer.step()
# Lets print a and b after the backward pass for certain epochs only.
if(epoch%100==0):
print("a and b after backward pass" , a,b)
# We reset it to zero after each epoch
#a.grad.zero_()
#b.grad.zero_()
# We will use optimizer to reset the gradients to zero.
optimizer.zero_grad()
# Lets print a and b after the backward pass and after reset to zero for certain epochs only.
if(epoch%100==0):
print("a and b after backward pass after resetting grad to zero" , a,b)
torch.manual_seed(42)
a = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
b = torch.randn(1, requires_grad=True, dtype=torch.float, device=device)
a, b
lr = 1e-1
n_epochs = 1000
# Define a SGD optimizer to update the parameters
optimizer = optim.SGD([a, b], lr=lr)
loss_func = nn.MSELoss(reduction='mean')
# reduction = 'mean' basically says you aggregate the losses by mean.
# reduction can also be sum in which case it is SSE. Sum Squared Error.
for epoch in range(n_epochs):
yhat = a + b * x_train_tensor
# error = y_train_tensor - yhat
#loss = (error ** 2).mean()
# We commented the loss calculation and error calculation since
# both will be taken care of by loss_func defined above.
loss = loss_func(y_train_tensor, yhat)
# Lets print a and b before the backward pass for certain epochs only.
if(epoch%100==0):
print("epoch number =", epoch)
print("a and b before backward pass" , a,b)
# We will not do this now like we did in Numpy notebook
# - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb
# a_grad = -2 * error.mean()
# b_grad = -2 * (x_tensor * error).mean()
# We will tell PyTorch to do the back progagation to the beginning from the specified loss which is at the end.
loss.backward()
# Let's check the computed gradients...
# Lets print the gradients for certain epochs only.
if(epoch%100==0):
print("a's gradient = " , a.grad)
print("b's gradient = " , b.grad)
# Let's try to update the parameters with the formula we know - parameters = parameters - learning rate* gradient(parameters wrto Loss function)
# We are telling Pytorch not to calculate gradients for steps within the "with"
#with torch.no_grad():
# a-= lr * a.grad
# b.sub_(lr * b.grad)
# We will use optimizer to update the parameters.
optimizer.step()
# Lets print a and b after the backward pass for certain epochs only.
if(epoch%100==0):
print("a and b after backward pass" , a,b)
# We reset it to zero after each epoch
#a.grad.zero_()
#b.grad.zero_()
# We will use optimizer to reset the gradients to zero.
optimizer.zero_grad()
# Lets print a and b after the backward pass and after reset to zero for certain epochs only.
if(epoch%100==0):
print("a and b after backward and zeroing grad" , a,b)
init(self) - As the name says this is for the initialization. Setting up of the parameters. In this case of LR, we have a and bclass MyLRModel(nn.Module):
def __init__(self):
super().__init__()
# To make "a" and "b" real parameters of the model, we need to wrap them in nn.Parameter
self.a = nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float))
self.b = nn.Parameter(torch.randn(1, requires_grad=True, dtype=torch.float))
def forward(self, x):
# Computes the predictions and in this case it is a + bx
return self.a + self.b * x
#Keeping the seed same as before. This has to be done before defining the model.
torch.manual_seed(42)
# Now we can create a model and send it to the device. Model should be on the same device as the parameters.
model = MyLRModel().to(device)
# To print the values of a and b , we use state_dict() method.
print(model.state_dict())
lr = 1e-1
n_epochs = 1000
# Define a SGD optimizer to update the parameters. Note that parameters are now passed via model.parameters() and not manually.
optimizer = optim.SGD(model.parameters(), lr=lr)
loss_func = nn.MSELoss(reduction='mean')
# reduction = 'mean' basically says you aggregate the losses by mean.
# reduction can also be sum in which case it is SSE. Sum Squared Error.
for epoch in range(n_epochs):
# This is to tell Pytorch that model should be in training mode at this time and not evaluation mode.
# The model behavior might change in different modes.
model.train()
yhat = model(x_train_tensor)
loss = loss_func(y_train_tensor, yhat)
loss.backward()
optimizer.step()
optimizer.zero_grad()
# To print the values of a and b after all epochs, we use state_dict() method.
print(model.state_dict())
yhat = model(x_train_tensor)
loss = loss_func(y_train_tensor, yhat)
loss.backward()
optimizer.step()
optimizer.zero_grad()
def make_train_step(model, loss_fn, optimizer):
# Builds function that performs a step in the train loop
def train_step(x, y):
# Sets model to training mode
model.train()
# Makes predictions
yhat = model(x)
# Computes loss
loss = loss_func(y, yhat)
# Computes gradients
loss.backward()
# Updates parameters and zeroes gradients
optimizer.step()
optimizer.zero_grad()
# Returns the loss
return loss.item()
# Returns the function that will be called inside the train loop
return train_step
# Creates the train_step function for our model, loss function and optimizer
train_step = make_train_step(model, loss_func, optimizer)
losses = []
# For each epoch...
for epoch in range(n_epochs):
# Performs one train step and returns the corresponding loss
loss = train_step(x_train_tensor, y_train_tensor)
losses.append(loss)
# Checks model's parameters
print(model.state_dict())
from torch.utils.data import Dataset, TensorDataset
class CustomDataset(Dataset):
def __init__(self, x_tensor, y_tensor):
self.x = x_tensor
self.y = y_tensor
def __getitem__(self, index):
return (self.x[index], self.y[index])
def __len__(self):
return len(self.x)
x_train_tensor = torch.from_numpy(x_train).float()
y_train_tensor = torch.from_numpy(y_train).float()
train_data = CustomDataset(x_train_tensor, y_train_tensor)
print(train_data[0])
type(train_data)
train_data = TensorDataset(x_train_tensor, y_train_tensor)
print(train_data[0])
type(train_data)
len(x_train_tensor) , type(x_train_tensor), len(train_data), type(train_data)
len(x_val), type(x_val)
x_valid_tensor = torch.from_numpy(x_val).float()
y_valid_tensor = torch.from_numpy(y_val).float()
valid_data = TensorDataset(x_valid_tensor, y_valid_tensor)
len(x_valid_tensor), type(x_valid_tensor), len(valid_data), type(valid_data)
train_data.to(device)
from torch.utils.data import DataLoader
train_loader = DataLoader(dataset=train_data, batch_size=16, shuffle=True)
next(iter(train_loader))
len(next(iter(train_loader.batch_sampler)))
train_loader.batch_size
train_loader.dataset[0]
train_data[0]
losses = []
train_step = make_train_step(model, loss_func, optimizer)
for epoch in range(n_epochs):
for x_batch, y_batch in train_loader:
# The dataset, dataloader and its mini-batches all resides in the CPU for now.
# Lets send those mini-batches to the GPU in below steps
# Model is already in GPU
x_batch = x_batch.to(device)
y_batch = y_batch.to(device)
loss = train_step(x_batch, y_batch)
losses.append(loss)
print(model.state_dict())
valid_loader = DataLoader(dataset=valid_data, batch_size=16, shuffle=True)
losses = []
val_losses = []
train_step = make_train_step(model, loss_func, optimizer)
for epoch in range(n_epochs):
for x_batch, y_batch in train_loader:
x_batch = x_batch.to(device)
y_batch = y_batch.to(device)
loss = train_step(x_batch, y_batch)
losses.append(loss)
with torch.no_grad():
for x_val, y_val in valid_loader:
x_val = x_val.to(device)
y_val = y_val.to(device)
model.eval()
yhat = model(x_val)
val_loss = loss_func(y_val, yhat)
val_losses.append(val_loss.item())
print(model.state_dict())
len(val_losses)
val_losses[1990:]
len(losses)
losses[4990:]
This is the final prediction of the model which is pretty accurate, slightly more accurate than our numpy model - https://github.com/manjunathrgithub/Simple-LR-Model-with-Numpy/blob/master/LR_Using_Numpy.ipynb