Shape of tensor changes after slicing. RuntimeError: stack expects each tensor to be equal size, but got [32, 1] at entry 0 and [32, 0] at entry 1

Umair_Javaid · May 2, 2020, 10:54am

I have a very large tensor of shape (512,3,224,224). I input it to model in batches of 32 and I then save the scores corresponding to the target label which is 2. in each iteration, after every slice, the shape of scores changes. Which leads to the following error. What am I doing wrong and how to fix it.
label = torch.ones(1)*2

def sub_forward(self, x):
    x = self.vgg16(x)
    x = self.bn1(x)
    x = self.linear1(x)
    x = self.linear2(x)
    return x


def get_scores(self, imgs, targets):
    b, _, _, _ = imgs.shape
    batch_size = 32
    total_scores = []
    for i in range(0, b, batch_size):
      scores = self.sub_forward(imgs[i:i+batch_size,:,:,:])
      scores = F.softmax(scores)
      labels = targets[i:i+batch_size]
      labels = labels.long()
      scores = scores[:,labels]
      print(i," scores: ", scores)
      total_scores.append(scores)
      print(i," total_socres: ", total_scores)
    total_scores = torch.stack(total_scores)
    return scores

0  scores:  tensor([[0.0811],
        [0.0918],
        [0.0716],
        [0.1680],
        [0.1689],
        [0.1319],
        [0.1556],
        [0.2966],
        [0.0913],
        [0.1238],
        [0.1480],
        [0.1215],
        [0.2524],
        [0.1283],
        [0.1603],
        [0.1282],
        [0.2668],
        [0.1146],
        [0.2043],
        [0.2475],
        [0.0865],
        [0.1869],
        [0.0860],
        [0.1979],
        [0.1677],
        [0.1983],
        [0.2623],
        [0.1975],
        [0.1894],
        [0.3299],
        [0.1970],
        [0.1094]], device='cuda:0')
0  total_socres:  [tensor([[0.0811],
        [0.0918],
        [0.0716],
        [0.1680],
        [0.1689],
        [0.1319],
        [0.1556],
        [0.2966],
        [0.0913],
        [0.1238],
        [0.1480],
        [0.1215],
        [0.2524],
        [0.1283],
        [0.1603],
        [0.1282],
        [0.2668],
        [0.1146],
        [0.2043],
        [0.2475],
        [0.0865],
        [0.1869],
        [0.0860],
        [0.1979],
        [0.1677],
        [0.1983],
        [0.2623],
        [0.1975],
        [0.1894],
        [0.3299],
        [0.1970],
        [0.1094]], device='cuda:0')]
32  scores:  tensor([], device='cuda:0', size=(32, 0))
32  total_socres:  [tensor([[0.0811],
        [0.0918],
        [0.0716],
        [0.1680],
        [0.1689],
        [0.1319],
        [0.1556],
        [0.2966],
        [0.0913],
        [0.1238],
        [0.1480],
        [0.1215],
        [0.2524],
        [0.1283],
        [0.1603],
        [0.1282],
        [0.2668],
        [0.1146],
        [0.2043],
        [0.2475],
        [0.0865],
        [0.1869],
        [0.0860],
        [0.1979],
        [0.1677],
        [0.1983],
        [0.2623],
        [0.1975],
        [0.1894],
        [0.3299],
        [0.1970],
        [0.1094]], device='cuda:0'), tensor([], device='cuda:0', size=(32, 0))]

`

RuntimeError: stack expects each tensor to be equal size, but got [32, 1] at entry 0 and [32, 0] at entry 1

`

InnovArul · May 2, 2020, 11:04am

It seems that this line gives 0 results in 2nd iteration. What does labels contain?

Umair_Javaid · May 2, 2020, 11:07am

I forward label to get_scores function
label = torch.ones(1)*2
get_scores(self, imgs, label)

InnovArul · May 2, 2020, 11:11am

Something is not right.

You won’t be able to index targets if you define label as label = torch.ones(1)*2.

Umair_Javaid · May 2, 2020, 11:12am

well that works fine for me. later I will be having a much larger tensor of targets

InnovArul · May 2, 2020, 11:14am

Interesting.

x = torch.tensor(1)*2 
x[0]

To me, this gives invalid index of a 0-dim tensor. Use tensor.item() to convert a 0-dim tensor to a Python number in pytorch 1.3.1.

Umair_Javaid · May 2, 2020, 11:19am

Interesting indeed.
For the following code I get no error

label = torch.ones(1)*2
print(label[32:64])

It gives the following result:
`

tensor()

`
I am using google colab atm

InnovArul · May 2, 2020, 11:21am

This explains why there are 0 results.

can you check the pytorch version?
print(torch.__version__)

Umair_Javaid · May 2, 2020, 11:22am

PyTorch Version:  1.5.0+cu101
Torchvision Version:  0.6.0+cu101

(What to do now?)

Umair_Javaid · May 2, 2020, 11:41am

Are there any alternatives to this?

InnovArul · May 2, 2020, 1:51pm

@ptrblck any suggestions?

ptrblck · May 3, 2020, 12:45am

Slicing “out of bounds” will return an empty tensor, which is expected.
It seems you are forwarding targets, not labels to get_scores and calculate labels as:

labels = targets[i:i+batch_size]

Could you check the shape of targets before the slicing operation and labels afterwards?

Umair_Javaid · May 3, 2020, 2:33am

def get_scores(self, imgs, targets):
    b, _, _, _ = imgs.shape
    batch_size = 32
    total_scores = []
    for i in range(0, b, batch_size):
      scores = self.sub_forward(imgs[i:i+batch_size,:,:,:])
      scores = F.softmax(scores)
      print("targets: ",targets)
      labels = targets[i:i+batch_size]
      print("labels: ",labels)
      labels = labels.long()
      scores = scores[:,labels]
      print(i," scores: ", scores)
      total_scores.append(scores)
      print(i," total_socres: ", total_scores)
    total_scores = torch.stack(total_scores)
    return scores


targets:  tensor([2.])
labels:  tensor([2.])
/usr/local/lib/python3.6/dist-packages/ipykernel_launcher.py:38: UserWarning: Implicit dimension choice for softmax has been deprecated. Change the call to include dim=X as an argument.
0  scores:  tensor([[0.5671],
        [0.5106],
        [0.5338],
        [0.5548],
        [0.5764],
        [0.5593],
        [0.6007],
        [0.5891],
        [0.5425],
        [0.6006],
        [0.6386],
        [0.5034],
        [0.5746],
        [0.5831],
        [0.5794],
        [0.6206],
        [0.2900],
        [0.4843],
        [0.6075],
        [0.4473],
        [0.5612],
        [0.2937],
        [0.6049],
        [0.5867],
        [0.3828],
        [0.4014],
        [0.4806],
        [0.5595],
        [0.5749],
        [0.5216],
        [0.3748],
        [0.5434]], device='cuda:0')
0  total_socres:  [tensor([[0.5671],
        [0.5106],
        [0.5338],
        [0.5548],
        [0.5764],
        [0.5593],
        [0.6007],
        [0.5891],
        [0.5425],
        [0.6006],
        [0.6386],
        [0.5034],
        [0.5746],
        [0.5831],
        [0.5794],
        [0.6206],
        [0.2900],
        [0.4843],
        [0.6075],
        [0.4473],
        [0.5612],
        [0.2937],
        [0.6049],
        [0.5867],
        [0.3828],
        [0.4014],
        [0.4806],
        [0.5595],
        [0.5749],
        [0.5216],
        [0.3748],
        [0.5434]], device='cuda:0')]
targets:  tensor([2.])
labels:  tensor([])
32  scores:  tensor([], device='cuda:0', size=(32, 0))
32  total_socres:  [tensor([[0.5671],
        [0.5106],
        [0.5338],
        [0.5548],
        [0.5764],
        [0.5593],
        [0.6007],
        [0.5891],
        [0.5425],
        [0.6006],
        [0.6386],
        [0.5034],
        [0.5746],
        [0.5831],
        [0.5794],
        [0.6206],
        [0.2900],
        [0.4843],
        [0.6075],
        [0.4473],
        [0.5612],
        [0.2937],
        [0.6049],
        [0.5867],
        [0.3828],
        [0.4014],
        [0.4806],
        [0.5595],
        [0.5749],
        [0.5216],
        [0.3748],
        [0.5434]], device='cuda:0'), tensor([], device='cuda:0', size=(32, 0))]
targets:  tensor([2.])
labels:  tensor([])
64  scores:  tensor([], device='cuda:0', size=(32, 0))
64  total_socres:  [tensor([[0.5671],
        [0.5106],
        [0.5338],
        [0.5548],
        [0.5764],
        [0.5593],
        [0.6007],
        [0.5891],
        [0.5425],
        [0.6006],
        [0.6386],
        [0.5034],
        [0.5746],
        [0.5831],
        [0.5794],
        [0.6206],
        [0.2900],
        [0.4843],
        [0.6075],
        [0.4473],
        [0.5612],
        [0.2937],
        [0.6049],
        [0.5867],
        [0.3828],
        [0.4014],
        [0.4806],
        [0.5595],
        [0.5749],
        [0.5216],
        [0.3748],
        [0.5434]], device='cuda:0'), tensor([], device='cuda:0', size=(32, 0)), tensor([], device='cuda:0', size=(32, 0))]
targets:  tensor([2.])
labels:  tensor([])
96  scores:  tensor([], device='cuda:0', size=(32, 0))
96  total_socres:  [tensor([[0.5671],
        [0.5106],
        [0.5338],
        [0.5548],
        [0.5764],
        [0.5593],
        [0.6007],
        [0.5891],
        [0.5425],
        [0.6006],
        [0.6386],
        [0.5034],
        [0.5746],
        [0.5831],
        [0.5794],
        [0.6206],
        [0.2900],
        [0.4843],
        [0.6075],
        [0.4473],
        [0.5612],
        [0.2937],
        [0.6049],
        [0.5867],
        [0.3828],
        [0.4014],
        [0.4806],
        [0.5595],
        [0.5749],
        [0.5216],
        [0.3748],
        [0.5434]], device='cuda:0'), tensor([], device='cuda:0', size=(32, 0)), tensor([], device='cuda:0', size=(32, 0)), tensor([], device='cuda:0', size=(32, 0))]

Umair_Javaid · May 3, 2020, 2:36am

Should I use an if condition before slicing to see if I’m slicing out of bounds?

ptrblck · May 3, 2020, 2:59am

I’m not sure if a condition will save you, since you expect targets to have at least the length b+batch_size, while it seems to have a single element.
I would recommend to take another look, how targets is defined and why it’s smaller than you expect.

Umair_Javaid · May 3, 2020, 4:40am

you’re right. In my code, I’m repeating input images but I didn’t do that for the labels. Thank you so much for your time. Can you look at the other question I posted after this