For example i have batch of [batch_size, lens, features] and [batch_size,] for x and x_lens.
So masking should be like this:
x = torch.rand(32, 1000, 512)
lens = (torch.rand(32,)*1000).long()
power=3
masks=10
x = x.clone()
batch_size, length, features = x.size()
T = lens // power // masks
mean = (x.detach().sum(1) / lens.unsqueeze(-1)).unsqueeze(1).expand(-1, length, -1)
mask_start = (torch.rand((batch_size, masks), device=lens.device)*(lens-T).unsqueeze(1)).long()
mask_end = mask_start + (torch.rand((batch_size, masks), device=lens.device)*T.unsqueeze(1)).long()
mask = torch.arange(0, length, device=lens.device).unsqueeze(0).expand(batch_size, -1)
# here im stuck
mask = (mask >= mask_start) & (mask < mask_end)
x[mask] = mean[mask]
Im getting
RuntimeError: The size of tensor a (1000) must match the size of tensor b (10) at non-singleton dimension 1
I cant figure out how to apply all these masks.