I am trying to find the fastest/most efficient way to create a tensor to use for a custom dropout layer. I have not being able to figure out how to do this other than in a python loop to set the index of the tensor to 0 then multiply with the input.
More specifically, I want to set X number of random elements to 0 within a Y sized “group”. Example: 1d tensor of 10 elements, drop/group =2, groupSize=5: then 2 random indices in the the first half are 0 and 2 random indices from the second half are 0. Below is the code that I have working, but is terribly slow.
batch,feature = x.size()
scalar_mask = torch.tensor([self.scalar for _ in range(feature)])
for i in range(0,feature,self.group_size):
masked_count=0
while masked_count < self.drop_per_group:
index = random.randint(0, self.group_size-1)
if scalar_mask[i+index] !=0:
scalar_mask[i+index]=0
masked_count+=1
x = torch.multiply(x,scalar_mask)