How could I slice a tensor with various dimensions?

1

Hi,

I could slice my tensor like this: label[ignore[:,0],:,ignore[:,1],ignore[:,2]] = 0, this code works only when the ignore tensor has shape of (N, 3). When ignore has shape of (N, 4), it will be sliced like this: label[ignore[:,0],:,ignore[:,1],ignore[:,2], ignore[3,:]] = 0.

How could I broadcast this operation to various shape such as (N, M) ?

2

You could use chunk:

label[ignore.chunk(M, dim=1)] = 0
2 Likes
3

Hi,

It does not work for me, my code is like this:

    def forward(self, logits, label):
        logs = self.log_softmax(logits)
        ignore = label.data==self.lb_ignore
        label[ignore] = 0
        lb_one_hot = logits.data.clone().zero_().scatter_(1, label.unsqueeze(1), 1)
        label = self.lb_pos * lb_one_hot + self.lb_neg * (1-lb_one_hot)
        ignore = ignore.nonzero()
        N, M = ignore.size()
        #  label[ignore[:,0],:,ignore[:,1],ignore[:,2]] = 0
        label[ignore.chunk(M, dim=1)] = 0

        if self.reduction == 'mean':
            loss = -torch.mean(torch.sum(logs*label, dim=1))
        elif self.reduction == 'none':
            loss = -torch.sum(logs*label, dim=1)
        return loss

Error message is:

  File "/home/zhangzy/build/miniconda/envs/py37/lib/python3.7/site-packages/torch/nn/modules/m
odule.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "label_smooth.py", line 32, in forward
    label[ignore.chunk(M, dim=1)] = 0
IndexError: index 3 is out of bounds for dimension 1 with size 3
4

Could you post the shapes of label and ignore as well as the values of ignore?
I’ve tried to test the method, and thought chunk would work if the manual approach using your code has worked before.

5

Hi,

The full code can be like this:

class LabelSmoothSoftmaxCE(nn.Module):
    def __init__(self,
            lb_pos = 0.9,
            lb_neg = 0.005,
            reduction = 'mean',
            lb_ignore = 255,
            *args, **kwargs):
        super(LabelSmoothSoftmaxCE, self).__init__()
        self.lb_pos = lb_pos
        self.lb_neg = lb_neg
        self.reduction = reduction
        self.lb_ignore = lb_ignore
        self.log_softmax = nn.LogSoftmax(1)

    def forward(self, logits, label):
        logs = self.log_softmax(logits)
        ignore = label.data==self.lb_ignore
        label[ignore] = 0
        lb_one_hot = logits.data.clone().zero_().scatter_(1, label.unsqueeze(1), 1)
        label = self.lb_pos * lb_one_hot + self.lb_neg * (1-lb_one_hot)
        ignore = ignore.nonzero()
        N, M = ignore.size()
        #  label[ignore[:,0],:,ignore[:,1],ignore[:,2]] = 0
        label[ignore.chunk(M, dim=1)] = 0is 

        if self.reduction == 'mean':
            loss = -torch.mean(torch.sum(logs*label, dim=1))
        elif self.reduction == 'none':
            loss = -torch.sum(logs*label, dim=1)
        return loss

if __name__ == '__main__':
    logits = torch.randn(2, 3, 4, 4)
    label = torch.randint(0, 3, (2, 4, 4))

    label[1, 2, 2] = 255
    label[1, 3, 2] = 255
    criteria_softmax = LabelSmoothSoftmaxCE(lb_pos=0.9, lb_neg=0.05, reduction='none')
    loss = criteria_softmax(logits, label)
    print(loss)

As can be seen, the input logits has size (2,3,4,4), and label has size (2,4,4), I am trying to make this in consistent with the requirement of nn.CrossEntropyLoss.

How could I cope with this problem please?

Edit:
I noticed that this implementation can enlarge the usage of memory during training, do I have better approaches to do it?

6

Ah sorry for the mistake! I didn’t see the colon in your indexing code.
This should work now:

a, *b = ignore.chunk(M, dim=1)
label[[a.view(-1, 1), torch.arange(label.size(1)), *b]] = 0
1 Like