Stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1

audio
1

Hi all, I have pre-processed my dataset to obtained three sets as train test and validation. The shapes and type of each of them are as follows.

Shape of X_train:  (3441, 7, 1, 128, 128)
type(X_train): numpy.ndarray
Shape of X_val:  (143, 7, 1, 128, 128)
type(X_val): numpy.ndarray
Shape of X_test:  (150, 7, 1, 128, 128)
type(X_test): numpy.ndarray

The class Dataset is created as given:

import torch
from torch.utils.data import Dataset

class Dataset(Dataset):
    def __init__(this, X=None, y=None,  mode="train"):
        this.mode = mode
        this.X = X
        if mode == "train":

            this.y = y
            

    def __len__(this):
        return this.X.shape[0]

    def __getitem__(this, idx):
        if this.mode == "train":
            return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])
        else:
            return torch.FloatTensor(this.X[idx])

Then I have use PytorchDataloader as

train_set = Dataset(X=X_train, y=Y_train,mode="train")
tr_loader = DL(train_set, batch_size=32, shuffle=True)

test_set  = Dataset(X=X_test, y=Y_test,mode="train")
ts_loader = DL(test_set, batch_size=32, shuffle=False)

Everything is working fine uptill now, (I think so :)). Then for training I used the code below.

verbose=True
Losses = []
Accuracies = []
epochs=10
DLS = {"train": tr_loader, "valid": ts_loader}

start_time = time.time()
for e in range(epochs):
    epochLoss = {"train": 0, "valid": 0}
    epochAccs = {"train": 0, "valid": 0}

    for phase in ["train", "valid"]:
        if phase == "train":
            model.train()
        else:
            model.eval()

        lossPerPass = []
        accuracy = []

        for X, y in DLS[phase]:
            X, y = X.to(device), y.to(device).view(-1)

            optimizer.zero_grad()
            alpha=1.0
            beta=1.0
            with torch.set_grad_enabled(phase == "train"):
                
                pred_emo= model(X)
                emotion_loss = criterion(pred_emo,y)
               
                total_loss = alpha*emotion_loss
                if phase == "train":
                    total_loss.backward()
                    optimizer.step()
            lossPerPass.append(total_loss.item())
            accuracy.append(accuracy_score(torch.argmax(torch.exp(pred_emo.detach().cpu()), dim=1), y.cpu()))
            torch.save(model.state_dict(),"E:/Python_On_All_Dataset/IEMO/long_codes/Kosta.jo/model_checkpoint_spec/Epoch_{}.pt".format(e+1))  
        epochLoss[phase] = np.mean(np.array(lossPerPass))
        epochAccs[phase] = np.mean(np.array(accuracy))
      
    Losses.append(epochLoss)
    Accuracies.append(epochAccs)



    if verbose:
        print("Epoch : {} | Train Loss : {:.5f} | Valid Loss : {:.5f} \
| Train Accuracy : {:.5f} | Valid Accuracy : {:.5f}".format(e + 1, epochLoss["train"], epochLoss["valid"],
                                                        epochAccs["train"], epochAccs["valid"]))

Then I am getting this error traceback:

File “”, line 21, in
for X, y in DLS[phase]:

File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 517, in next
data = self._next_data()

File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py”, line 557, in _next_data
data = self._dataset_fetcher.fetch(index) # may raise StopIteration

File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\fetch.py”, line 47, in fetch
return self.collate_fn(data)

File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in default_collate
return [default_collate(samples) for samples in transposed]

File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 83, in
return [default_collate(samples) for samples in transposed]

File “C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data_utils\collate.py”, line 55, in default_collate
return torch.stack(batch, 0, out=out)

RuntimeError: stack expects each tensor to be equal size, but got [0] at entry 0 and [3] at entry 1

Can someone comment on this, Is there any problem with class Dataset or in training. The tittle of the problem can be changed on suggestions.
please guide
Regards

2

I cannot reproduce the issue using:

X_train = torch.randn(3441, 7, 1, 128, 128)
y_train = torch.randint(0, 10, (3441,))

dataset = Dataset(X_train, y_train)
loader = DataLoader(dataset, batch_size=16)

for data, target in loader:
    print(data.shape)
    print(target.shape)

Could you compare my code snippet to yours and check, what the difference might be?

1 Like
3

@ptrblck sir my Y_train shape is (3441,), is this the problem? If yes what to do:
also when I run your code on my class dataset it says:

slice() cannot be applied to a 0-dim tensor. in line (in class Dataset)
return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])

I have changed the numpy array into tensor using this:

X_test=torch.from_numpy(X_test)

then I got an error in line:
return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])

TypeError: expected Float (got Double)
Sir plz tell me if I need to share the model also. I think the whole problem is in loading the dataset .

Regards

4

This shouldn’t be a problem, since my code snippet uses the same shape for y_train as well as your Dataset definition.

numpy uses float64 by default, so you might want to transform the tensor to float32 via:

X_test = torch.from_numpy(X_test).float()

A minimal and executable code snippet would be helpful to further debug the issue.

5

@ptrblck sir what about this? I have changed the data to float() and targets as long(). Now it showing the above error at
return torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])

6

I would try to use the original numpy array and converted into torch tensor here:

class Dataset(Dataset):
    def __getitem__(this, idx):
        if this.mode == "train":
            return torch.from_numpy(this.X[idx]).float(), torch.from_numpy(this.y[idx]).long()
        else:
            return torch.from_numpy(this.X[idx]).float()

Your original back tracing is interesting

It falls into the logic here https://github.com/pytorch/pytorch/blob/94cc681fc2e5218c10493938a8ca01272c3c6fc0/torch/utils/data/_utils/collate.py#L77-L84, which is not expected since you are returning tuple. Can you verify your output by printing out the data like the following code.

class Dataset(Dataset):
    def __getitem__(this, idx):
        if this.mode == "train":
            res =  torch.from_numpy(this.X[idx]).float(), torch.from_numpy(this.y[idx]).long()
            print(res)
            return res
        else:
            return torch.from_numpy(this.X[idx]).float()
7

@ptrblck @ejguan Sir
I am trying to explain the complete code here. The processed dataset is

Shape of X_train:  (1147, 7, 1, 128, 128)
Shape of X_val:  (143, 7, 1, 128, 128)
Shape of X_test:  (150, 7, 1, 128, 128)

b,t,c,h,w = X_train.shape
X_train = np.reshape(X_train, newshape=(b,-1))
X_train = scaler.fit_transform(X_train)
X_train = np.reshape(X_train, newshape=(b,t,c,h,w))

b,t,c,h,w = X_test.shape
X_test = np.reshape(X_test, newshape=(b,-1))
X_test = scaler.transform(X_test)
X_test = np.reshape(X_test, newshape=(b,t,c,h,w))

b,t,c,h,w = X_val.shape
X_val = np.reshape(X_val, newshape=(b,-1))
X_val = scaler.transform(X_val)
X_val = np.reshape(X_val, newshape=(b,t,c,h,w))

The model is

import torch
import torch.nn as nn
# BATCH FIRST TimeDistributed layer
class TimeDistributed(nn.Module):
    def __init__(self, module):
        super(TimeDistributed, self).__init__()
        self.module = module

    def forward(self, x):

        if len(x.size()) <= 2:
            return self.module(x)
        # squash samples and timesteps into a single axis
        elif len(x.size()) == 3: # (samples, timesteps, inp1)
            x_reshape = x.contiguous().view(-1, x.size(2))  # (samples * timesteps, inp1)
        elif len(x.size()) == 4: # (samples,timesteps,inp1,inp2)
            x_reshape = x.contiguous().view(-1, x.size(2), x.size(3)) # (samples*timesteps,inp1,inp2)
        else: # (samples,timesteps,inp1,inp2,inp3)
            x_reshape = x.contiguous().view(-1, x.size(2), x.size(3),x.size(4)) # (samples*timesteps,inp1,inp2,inp3)
            
        y = self.module(x_reshape)
        
        # we have to reshape Y
        if len(x.size()) == 3:
            y = y.contiguous().view(x.size(0), -1, y.size(1))  # (samples, timesteps, out1)
        elif len(x.size()) == 4:
            y = y.contiguous().view(x.size(0), -1, y.size(1), y.size(2)) # (samples, timesteps, out1,out2)
        else:
            y = y.contiguous().view(x.size(0), -1, y.size(1), y.size(2),y.size(3)) # (samples, timesteps, out1,out2, out3)
        return y
    
    
class HybridModel(nn.Module):
    def __init__(self,num_emotions):
        super().__init__()
        # conv block
        self.conv2Dblock = nn.Sequential(
            # 1. conv block
            TimeDistributed(nn.Conv2d(in_channels=1,
                                   out_channels=16,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(16)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=2, stride=2)),
            TimeDistributed(nn.Dropout(p=0.4)),
            # 2. conv block
            TimeDistributed(nn.Conv2d(in_channels=16,
                                   out_channels=32,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(32)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=4, stride=4)),
            TimeDistributed(nn.Dropout(p=0.4)),
            # 3. conv block
            TimeDistributed(nn.Conv2d(in_channels=32,
                                   out_channels=64,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(64)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=4, stride=4)),
            TimeDistributed(nn.Dropout(p=0.4)),
            # 4. conv block
            TimeDistributed(nn.Conv2d(in_channels=64,
                                   out_channels=128,
                                   kernel_size=3,
                                   stride=1,
                                   padding=1
                                  )),
            TimeDistributed(nn.BatchNorm2d(128)),
            TimeDistributed(nn.ReLU()),
            TimeDistributed(nn.MaxPool2d(kernel_size=4, stride=4)),
            TimeDistributed(nn.Dropout(p=0.4))
        )
        # LSTM block
        hidden_size = 64
        self.lstm = nn.LSTM(input_size=128,hidden_size=hidden_size,bidirectional=False, batch_first=True) 
        self.dropout_lstm = nn.Dropout(p=0.3)
        # Linear softmax layer
        self.out_linear = nn.Linear(hidden_size,num_emotions)
    def forward(self,x):
        conv_embedding = self.conv2Dblock(x)
        conv_embedding = torch.flatten(conv_embedding, start_dim=2) # do not flatten batch dimension and time
        lstm_embedding, (h,c) = self.lstm(conv_embedding)
        lstm_embedding = self.dropout_lstm(lstm_embedding)
        # lstm_embedding (batch, time, hidden_size)
        lstm_output = lstm_embedding[:,-1,:] 
        output_logits = self.out_linear(lstm_output)
        output_softmax = nn.functional.softmax(output_logits,dim=1)
        return output_logits, output_softmax

The train test definition are

def loss_fnc(predictions, targets):
    return nn.CrossEntropyLoss()(input=predictions,target=targets)    

def make_train_step(model, loss_fnc, optimizer):
    def train_step(X,Y):
        # set model to train mode
        model.train()
        # forward pass
        output_logits, output_softmax = model(X)
        predictions = torch.argmax(output_softmax,dim=1)
        accuracy = torch.sum(Y==predictions)/float(len(Y))
        # compute loss
        loss = loss_fnc(output_logits, Y)
        # compute gradients
        loss.backward()
        # update parameters and zero gradients
        optimizer.step()
        optimizer.zero_grad()
        return loss.item(), accuracy*100
    return train_step


def make_validate_fnc(model,loss_fnc):
    def validate(X,Y):
        with torch.no_grad():
            model.eval()
            output_logits, output_softmax = model(X)
            predictions = torch.argmax(output_softmax,dim=1)
            accuracy = torch.sum(Y==predictions)/float(len(Y))
            loss = loss_fnc(output_logits,Y)
        return loss.item(), accuracy*100, predictions
    return validate

and final training previously was

EPOCHS=700
DATASET_SIZE = X_train.shape[0]
BATCH_SIZE = 32
device = 'cuda' if torch.cuda.is_available() else 'cpu'
print('Selected device is {}'.format(device))
model = HybridModel(num_emotions=len(EMOTIONS)).to(device)
print('Number of trainable params: ',sum(p.numel() for p in model.parameters()) )
OPTIMIZER = torch.optim.SGD(model.parameters(),lr=0.01, weight_decay=1e-3, momentum=0.8)

train_step = make_train_step(model, loss_fnc, optimizer=OPTIMIZER)
validate = make_validate_fnc(model,loss_fnc)
losses=[]
val_losses = []
for epoch in range(EPOCHS):
    # schuffle data
    ind = np.random.permutation(DATASET_SIZE)
    X_train = X_train[ind,:,:,:,:]
    Y_train = Y_train[ind]
    epoch_acc = 0
    epoch_loss = 0
    iters = int(DATASET_SIZE / BATCH_SIZE)
    for i in range(iters):
        batch_start = i * BATCH_SIZE
        batch_end = min(batch_start + BATCH_SIZE, DATASET_SIZE)
        actual_batch_size = batch_end-batch_start
        X = X_train[batch_start:batch_end,:,:,:,:]
        Y = Y_train[batch_start:batch_end]
        X_tensor = torch.tensor(X,device=device).float()
        Y_tensor = torch.tensor(Y, dtype=torch.long,device=device)
        loss, acc = train_step(X_tensor,Y_tensor)
        epoch_acc += acc*actual_batch_size/DATASET_SIZE
        epoch_loss += loss*actual_batch_size/DATASET_SIZE
        print(f"\r Epoch {epoch}: iteration {i}/{iters}",end='')
    X_val_tensor = torch.tensor(X_val,device=device).float()
    Y_val_tensor = torch.tensor(Y_val,dtype=torch.long,device=device)
    val_loss, val_acc, _ = validate(X_val_tensor,Y_val_tensor)
    losses.append(epoch_loss)
    val_losses.append(val_loss)
    print('')
    print(f"Epoch {epoch} --> loss:{epoch_loss:.4f}, acc:{epoch_acc:.2f}%, val_loss:{val_loss:.4f}, val_acc:{val_acc:.2f}%")

Now this last step I need to change as the dataset is huge for my GPU so I m suggested to use pytorch dataloaders. I hope I am clear now.
Now along with previous dataset class whatelse I need to change please suggest.

Regards

8

My suggestion when you start to use DataLoader. You should use it without worker (by setting num_workers=0) to validate your dataset. Then, you can start to use multiple workers to accelerate your data loading speed.

9

@ejguan sir where is the option to set num_workers??

I have tried without data augmentation too, but still getting same error

IndexError:: slice() cannot be applied to a 0-dim tensor.

10

It’s an argument of DataLoader. (torch.utils.data — PyTorch 2.1 documentation)

Can you please copy the whole traceback of this Error?

1 Like
11

@ejguan Sir , I have made the change by putting num_workers=0 also, that made no difference here. Sir Now I think this whole code is executable, Plz check once.
This is the complete traceback

  File "E:\Python_On_All_Dataset\IEMO\long_codes\Kosta.jo\stacked _cnn.py", line 471, in <module>
    for X, y in DLS[phase]:

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 517, in __next__
    data = self._next_data()

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\dataloader.py", line 557, in _next_data
    data = self._dataset_fetcher.fetch(index)  # may raise StopIteration

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in fetch
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "C:\Users\krishna\Anaconda3\lib\site-packages\torch\utils\data\_utils\fetch.py", line 44, in <listcomp>
    data = [self.dataset[idx] for idx in possibly_batched_index]

  File "E:\Python_On_All_Dataset\IEMO\long_codes\Kosta.jo\Dataset.py", line 26, in __getitem__
    rec= torch.FloatTensor(this.X[idx]), torch.LongTensor(this.y[idx])

IndexError: slice() cannot be applied to a 0-dim tensor.
12

The problem comes from here:

Please try to replace torch.FloatTensor(this.X[idx]) by torch.tensor(this.X[idx], dtype=torch.float) and replace torch.LongTensor(this.y[idx]) by torch.tensor(this.y[idx], dtype=torch.long)

13

Thanks @ejguan I have tried this. Now how to replace this training part with dataloader? Now I am using the code below for model training.

verbose=True
Losses = []
Accuracies = []
epochs=10
DLS = {"train": tr_loader, "valid": ts_loader}

start_time = time.time()
for e in range(epochs):
    epochLoss = {"train": 0, "valid": 0}
    epochAccs = {"train": 0, "valid": 0}

    for phase in ["train", "valid"]:
        if phase == "train":
            model.train()
        else:
            model.eval()

        lossPerPass = []
        accuracy = []

        for X, y in DLS[phase]:
            X, y = X.to(device), y.to(device).view(-1)

            optimizer.zero_grad()
            alpha=1.0
            beta=1.0
            with torch.set_grad_enabled(phase == "train"):
                
                pred_emo= model(X)
                emotion_loss = criterion(pred_emo,y)
                
                total_loss = alpha*emotion_loss
                if phase == "train":
                    total_loss.backward()
                    optimizer.step()
            lossPerPass.append(total_loss.item())
            accuracy.append(accuracy_score(torch.argmax(torch.exp(pred_emo.detach().cpu()), dim=1), y.cpu()))
            torch.save(model.state_dict(),"E:/Python_On_All_Dataset/IEMO/long_codes/Kosta.jo/model_checkpoint_spec/Epoch_{}.pt".format(e+1))  
        epochLoss[phase] = np.mean(np.array(lossPerPass))
        epochAccs[phase] = np.mean(np.array(accuracy))
        # Epoch Checkpoint // All or Best
    Losses.append(epochLoss)
    Accuracies.append(epochAccs)

    # if scheduler:  # or use, if scheduler_1 or scheduler_2: // Use correct call method
    #     scheduler.step(epochLoss["valid"])
    #     #scheduler.step()

    if verbose:
        print("Epoch : {} | Train Loss : {:.5f} | Valid Loss : {:.5f} \
| Train Accuracy : {:.5f} | Valid Accuracy : {:.5f}".format(e + 1, epochLoss["train"], epochLoss["valid"],
                                                        epochAccs["train"], epochAccs["valid"]))

print("Time Taken [{} Epochs] : {:.2f} minutes".format(epochs, (time() - start_time) / 60))

print("Training Complete")

But in line

 emotion_loss = criterion(pred_emo,y)

it throws attribute error : ‘tuple’ object has no attribute 'log_softmax’
I know the model returning two values, as in class HybridModel()

return output_logits, output_softmax
How can I resolve this. Also am I using correct code for training sir?
As previously without dataloaders. There three functions are used for training validation etc.

14

This is occuring because your model’s forward gives u two outputs output_logits, output_softmax , the loss criterion which is crossnetropyloss here expects one of those as a tensor but currently its a tuple just change the line

pred_emo= model(X)

to

logits,softmax = model(X)

and pass accordingly to ur criterion and u r good to go

15

That’s done , Thank you
But My main question is I want to replicate the model training as before data loaders, How it is to be done, that needs a high level understanding of pytorch training probably.

16

Can you tell what do you mean by “replicate the model training as before data loaders”?

17

Hi @nikhil6041 if you read the querry since starting you may get it. But lemme brief you again.
That code was without dalaloaders but not working for me due to cuda memory issues, the whole code is here before loaders above. Then I used pytorch dataloaders, now I want the same training as before. If you go through previous code here you gt the idea for sure.

Thanks

18

@krishna511 I dont see much change in the difference you are training your model by the earlier approach for the expected gpu error there r several ways u can avoid it for example by calling

torch.cuda.empty_cache()

after each epoch will help you probably.

19

No @nikhil6041 I set batch size of 1,its not executing even for one epoch then how to empty cashe

20

@krishna511 okay can u paste the entire code snippet which is throwing errors here once as a reply?