Getting "/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [127,0,0] Assertion `srcIndex < srcSelectDimSize` failed." error while training

1

I am training a model where I am using indexing to pull out feature maps. The dimension of feature maps that I get from the backbone is variable, but every time I pool the same number of feature maps. I am getting following error:

dexSelectSmallIndex: block: [99,0,0], thread: [64,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [65,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [66,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [67,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [68,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [69,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [70,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [71,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [72,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [73,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [74,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [75,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [76,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [77,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [78,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [79,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [80,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [81,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [82,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [83,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [84,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [85,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [86,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [87,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [88,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [89,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [90,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [91,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [92,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [93,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [94,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [99,0,0], thread: [95,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [96,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [97,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [98,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [99,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [100,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [101,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [102,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [103,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [104,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [105,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [106,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [107,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [108,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [109,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [110,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [111,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [112,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [113,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [114,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [115,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [116,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [117,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [118,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [119,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [120,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [121,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [122,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [123,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [124,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [125,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [126,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [11,0,0], thread: [127,0,0] Assertion srcIndex < srcSelectDimSize failed.
R18_look_at_particular_patch: 1/1000: 0%| | 0/1844 [00:11<?, ?it/s]
Traceback (most recent call last):
File “main_2.py”, line 174, in
epoch_losses.update(train_loss.item(), len(x))
RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

How to resolve it?

2

The error is raised by a failing indexing operation. Rerun your code via CUDA_LAUNCH_BLOCKING=1 to isolate the failing line of code and fix the values in the index tensor.

3

Hi! Thanks for the reply. I tried that but I am getting a similar error. It seems that it is related to indexing.

Error tail part:

/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [49,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [50,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [51,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [52,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [53,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [54,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [55,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [56,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [57,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [58,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [59,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [60,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [61,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [62,0,0] Assertion srcIndex < srcSelectDimSize failed.
/pytorch/aten/src/ATen/native/cuda/Indexing.cu:646: indexSelectSmallIndex: block: [102,0,0], thread: [63,0,0] Assertion srcIndex < srcSelectDimSize failed.
R18_look_at_particular_patch: 1/1000: 0%| | 0/1844 [00:05<?, ?it/s]
Traceback (most recent call last):
File “main_2.py”, line 171, in
raw_outs = T_net(x)
File “/media/HDD8TB3/swapnil/env/sp_env/lib/python3.6/site-packages/torch/nn/modules/module.py”, line 1102, in _call_impl
return forward_call(*input, **kwargs)
File “/media/HDD8TB3/sourabh/deep_sight_deep_sup_class/gs_front_3_expos_combined/model.py”, line 154, in forward
selected_slices = torch.index_select(feats, 0, indices)
RuntimeError: CUDA error: device-side assert triggered

The code of the model which I am trying to train:

class ResNet_backbone_classifier_grades(nn.Module):
def init(self,patch_size=658,num_patches=8,num_feat_maps=512,n_classes=4,model=None):
super().init()
self.k = num_feat_maps
self.num_patches = num_patches
self.patch_size= patch_size

    self.model = model  

    ###### classification in grades 
    self.avg_pool_3d = nn.AdaptiveAvgPool3d((64, 2, 2))
    self.fc_phone_class_1 = nn.Linear(256,64)
    self.fc_phone_class_2 = nn.Linear(64,n_classes)
    self.relu = nn.ReLU()

def forward(self,x):
    ###########################################################
    x = x.view(-1,3,self.patch_size,self.patch_size)
    feats, out_1 = self.model(x)

    ###########################################################

    x = torch.softmax(out_1, dim=1)

    a, x = torch.max(x, dim=1)

    all_indices = x.reshape(-1,self.num_patches)

    final_feats = []

    print('All indices in current batch')
    print(all_indices)

    for i in range(len(all_indices)):

        indices = torch.nonzero(all_indices[i]).squeeze()
        print('Indices to pick in current sample: ')
        print(indices)
        if indices.numel() == 0:       
            x = feats[i*self.num_patches:(i+1)*self.num_patches,:,:,:].view(-1,21*21)
            channel_activations, indices = torch.max(x, dim=1)
            topk_values, topk_indices = torch.topk(channel_activations, self.k)
            topk_channels = x[topk_indices]
            x = topk_channels.reshape(-1,self.k,21,21)
            print('no defects')
            print(x.shape)

        else:
            feats = feats[i*self.num_patches:(i+1)*self.num_patches,:,:,:]
            print('Current sample feats dim: ')
            print(feats.shape)
            selected_slices = torch.index_select(feats, 0, indices)
            reshaped_map = selected_slices.view(-1, 21*21)  # Reshape to (-1, 441)
            channel_activations, indices = torch.max(reshaped_map, dim=1)  # Get max activations and indices along dim 1
            topk_values, topk_indices = torch.topk(channel_activations, self.k)
            topk_channels = reshaped_map[topk_indices]
            x = topk_channels.reshape(-1,self.k,21,21)
            print('defects')
            print(x.shape)

        final_feats.append(x)

    final_feats = torch.stack(final_feats,dim=0)
    x = self.avg_pool_3d(final_feats).reshape(final_feats.shape[0],-1)

    x = self.relu(self.fc_phone_class_1(x))
    out_2 = self.fc_phone_class_2(x)

    print(out_2.shape)

    return out_2

So basically it seems that the issue is related to torch.index_select. As mentioned earlier, I want to pool out the top 512 feature maps (activation-wise) from the variable number of feature maps which are always greater than 512.

4

Hey @ptrblck, I solved the issue. There was an issue related to the slicing of the tensor. It was a silly mistake related naming of variables. Thanks for your support.