For reference I try to implement this paper: https://openreview.net/pdf?id=HkO-PCmYl But for helping me you should not have to read it.
This is my forward pass in my nn.Module:
def forward(self, x): if self.training: alpha = Variable(torch.rand(x.size()).cuda()) else: alpha = Variable(torch.FloatTensor([0.5]).cuda()) p1 = self.path1(x) p2 = self.path2(x) alpha = torch.unsqueeze(alpha, dim=1) alpha = torch.unsqueeze(alpha, dim=2) alpha = torch.unsqueeze(alpha, dim=3) alpha = alpha.expand(p1.size()) x = alpha * p1 + (1 - alpha) * p2 return x
So two questions:
- Is there a better way to handle alpha? All the
- In the backward pass I want to handle the gradient with
alpha=0.5instead of the random
alphafrom the forward pass. How would I implement this? Maybe you can give a working implementation? I saw http://pytorch.org/docs/master/autograd.html#torch.autograd.Function, but I am not sure how to use it…