Writing a simple Gaussian noise layer in Pytorch

3

To speed things up, I was thinking it’d be nice if the variable had been pre-allocated onto the GPU, and instead we just fill in the values:

class DynamicGNoise(nn.Module):
    def __init__(self, shape, std=0.05):
        super().__init__()
        self.noise = Variable(torch.zeros(shape,shape).cuda())
        self.std   = std
        
    def forward(self, x):
        if not self.training: return x
        self.noise.data.normal_(0, std=self.std)
        
        print(x.size(), self.noise.size())
        return x + self.noise

Surprisingly, this did not work:

RuntimeError: size ‘[48 x 48]’ is invalid for input of with 8110080 elements at /home/uapatira/Desktop/pytorch-master/torch/lib/TH/THStorage.c:59

Eh? The output of the print statement is:

torch.Size([5, 704, 48, 48]) torch.Size([48, 48])

From what I understand of pytorch broadcasting semantics, this shouldn’t be a problem. In fact, if I try to: torch.FloatTensor([5, 176, 96, 96]) + torch.FloatTensor([96, 96]), it works just fine, but when I try to do .backwards(), it looks like the broadcasting breaks down?

Solved: Fixed using .expand() on self.noise.

1 Like