RuntimeError: cannot unsqueeze empty tensor

#1

I am a beginner here trying to use pytorch for classification where the features are all numeric values. I have been trying to follow other classification examples but they are either image or text which is not exactly what I want. I get the following error from the below code following a Kaggle Titanic Competition kernel:

RuntimeError: cannot unsqueeze empty tensor

import torch
import numpy as np
from torch.autograd import Variable
from torch.utils.data import Dataset, DataLoader
import pandas as pd
import torch.nn as nn
import torch.nn.functional as F


class Net(nn.Module):

    def __init__(self):
        super(Net, self).__init__()
        self.i1 = nn.Linear(74, 128)
        self.i1 = nn.Linear(128, 3)
        # 3 possible labels for now 

    def forward(self, x):
        x = self.i1(x)
        x = F.dropout(x, p=0.1)
        x = F.relu(x)
        x = self.i2(x)
        x = F.sigmoid(x)

        return x

class NumberDataset(Dataset):

    def __init__(self):
        self.data = pd.read_csv('./data.csv', header=0, usecols=fields)

        # Number of columns in data is 75
        # Number of columns for self.x or input is 74
        # Nunber of columns for self.y or target is 1
        self.x = torch.from_numpy(np.asarray(self.data.iloc[:, :-1])).type(torch.FloatTensor)
        self.y = torch.from_numpy(np.asarray(self.data.iloc[:, -1])).type(torch.LongTensor)
        self.len = len(self.data)

    def __getitem__(self, index):
        return self.x[index], self.y[index]

    def __len__(self):
        return self.len 
        # returns 640


if __name__ == '__main__':
    dataset = NumberDataset()
    train_loader = DataLoader(dataset=dataset, batch_size=128, shuffle=True)

    net = Net()
    batch_size = 128
    num_epochs = 10000
    learning_rate = 0.01
    batch_no = dataset.len // batch_size

    criterion = nn.CrossEntropyLoss()
    optimizer = torch.optim.Adam(net.parameters(), lr=learning_rate)

    for i, data in enumerate(train_loader, 0):
        # get the inputs and labels from data loader
        inputs, labels = data

        for epoch in range(num_epochs):
            if epoch % 5 == 0:
                print('Epoch {}'.format(epoch + 1))
            # Mini batch learning
            for i in range(batch_no):
                start = i * batch_size
                end = start + batch_size
                x_var = Variable(inputs[start:end])
                y_var = Variable(abels[start:end])
                # Forward + Backward + Optimize
                optimizer.zero_grad()
                ypred_var = net(x_var)
                loss = criterion(ypred_var, y_var)
                loss.backward()
                optimizer.step()

Where exactly am I going wrong? Is this approach right for a feed forward neural network?

#2

It’s just a guess, but it seems the error is coming from your batching.
Currently you are using a DataLoader with a batch_size of 128, i.e. in each iteration your DataLoader will provide a batch of 128 samples.
In this loop you are trying to apply your own batching using the batch number.
However, inputs and labels will only contain 128 samples, not the whole dataset.
After the first valid batch, you will get an empty slice, since start and end will be too large.

If you use a Dataset and DataLoader you don’t have to implement batching yourself. The DataLoader will take care of it (even using multiprocessing to speed up the loading). You just have to implement the logic to load a single sample in your Dataset.

Try to remove the batching code in your training procedure. Also, as a small side note, Variables are deprecated and since 0.4.0 you can use tensors directly.
The loop over the DataLoader corresponds to one epoch, so you should move the epoch up:

for epoch in range(num_epochs):
    for i, data in enumerate(train_loader):
      inputs, labels = data
      optimizer.zero_grad()
      ypred = net(inputs)
      ...

    if epoch%5 == 0:
      print(...)
#3

Thank you that solved that error. However, when I print out the loss it does seem to change. Is everything else with my logic sound ?

#4

Since you are training your model, the loss should decrease. Is that the case or how does the loss change?

#5

I’ve learned that the loss should go down but in my case it prints the same value on each training epoch.

Example:

Train Epoch: 1 [128/640 (20%)] Loss: 159.35
Train Epoch: 1 [256/640 (40%)] Loss: 149.98
Train Epoch: 1 [384/640 (60%)] Loss: 146.86
Train Epoch: 1 [512/640 (80%)] Loss: 145.30
Train Epoch: 1 [640/640 (100%)] Loss: 144.37
Train Epoch: 2 [128/640 (20%)] Loss: 140.62
Train Epoch: 2 [256/640 (40%)] Loss: 140.62
Train Epoch: 2 [384/640 (60%)] Loss: 140.62
Train Epoch: 2 [512/640 (80%)] Loss: 140.62
Train Epoch: 2 [640/640 (100%)] Loss: 140.62

Also, if I decrease the batch size from 128 to 64, the loss starts lower at 70

#6

The loss is decreasing in the first epoch and gets stuck in the second.
Maybe your setup is not optimal for your data.
Try to overfit a small sample of your data, i.e. take just a few samples (just up to 10 samples for each class), and try to decrease the training error to approx. zero.
Also, your learning rate might be a bit too high, which often yields an approx. constant loss.

I don’t really like the fact that the loss changes that much regarding the batch size.
Are you seeing the same effect for other batch sizes, i.e. 0.25*batch_size == 0.25*loss?
This should usually not be the case, as the loss is averaged over the batch dimension using the default settings for the loss function.

#7

It is not the case that 0.25*batch_size == 0.25*loss. I will create another topic on the issue of the loss