Training loss is not decreasing below a specific value

vision
#1

I am trying to solve a problem that I found in deep learning with pytorch course on Udacity: “Predict whether a student will get selected or rejected by the university”.

But i am getting the training loss ~0.2000 every time. The loss is not falling below this value. Could you please go through the code to give me feedback?


import pandas as pd
import numpy as np
data = pd.read_csv('student_data.csv')


import matplotlib.pyplot as plt

def plot_points(data):
    X = np.array(data[['gre', 'gpa']])
    Y = np.array(data['admit'])
    admitted = X[np.argwhere(Y==1)]
    rejected = X[np.argwhere(Y==0)]
    plt.scatter([s[0][0] for s in rejected], [s[0][1] for s in rejected], color='red')
    plt.scatter([s[0][0] for s in admitted], [s[0][1] for s in admitted], color='cyan')
    plt.xlabel('Test (GRE)')
    plt.ylabel('Grades (GPA)')

plot_points(data)

data = data.drop('rank', axis=1)

# Normalizing the values
data['gre'] /= max(data['gre'])
data['gpa'] /= max(data['gpa'])



# splitting in train and test set
sample = np.random.choice(data.index, size=int(len(data)*0.9), replace=False)
train_data, test_data = data.iloc[sample], data.drop(sample)

train_features = train_data.drop('admit', axis=1)
train_labels = train_data['admit']

import torch
from torch import nn, optim
import torch.nn.functional as F 

# converting dataFrames into tensors
input_tensor = torch.from_numpy(train_features.values).type(torch.FloatTensor)
label_tensor = torch.from_numpy(train_labels.values).type(torch.FloatTensor)


class Net(nn.Module):
    def __init__(self):
        super(Net, self).__init__()
        
        self.fc1 = nn.Linear(2, 4)
        self.fc2 = nn.Linear(4, 1)
   
    def forward(self, x):
        
        x = F.relu(self.fc1(x))
        x = (self.fc2(x))
        return x

model = Net()

criteron = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr=0.003)

losses=[]
EPOCHS = 501
for e in range(EPOCHS):
    train_loss = 0
    
    output = model(input_tensor)
    loss = criteron(output, label_tensor.view(360, 1))
    
    train_loss += loss.item()
    losses.append(train_loss)
    optimizer.zero_grad()
    loss.backward()
    optimizer.step()
    
    if e%100==0:
        print('epoch {}: Training loss: {:.4f}'.format(e+1, train_loss))

plt.plot(losses)

here is how the data looks like.

#2

Just some random suggestions:

  1. can you put optimizer.zero_grad() before loss = criteron(output, label_tensor.view(360, 1)).
  2. set model.train() before you start the training process

let me know if this does not work.

#3

thank you for your reply mate!

But this won’t help because optimizer.zero_grad() is required before the gradient calculation i.e loss.backward(). So it wont make any difference.

setting model.train() also wont help because models are by default into train mode until we explicitly put it into evaluation mode.