How to initialize registered parameters seperately?

autograd
1

Hi ! I was trying to add new parameters a,b in every layer of my network as follows:

self.a = nn.Parameter(torch.ones([1,1],dtype=torch.double), requires_grad=True)       ###
self.b = nn.Parameter(torch.ones([1,1],dtype=torch.double)*torch.sqrt(torch.Tensor([self.dim]).double()), requires_grad=True)       ###
self.register_parameter('aa',self.a)                                                                 ###
self.register_parameter('bb',self.b)

I want to initialize them separately, say self.a=1, self.b = sqrt(dim). But as it turns out self.a and self.b are initialized with other parameters in my network: self.a = -1.74 self.b=0.79. Do anybody know how to
initialize self.a, self.b separately, avoid initialization with other parameters in the network?
Also, I find that self.aa, self.bb have different data types from self.a, self.b when using GPU. How to solve this?

2

Your code works, if I just wrap it in a custom nn.Module:

class MyModel(nn.Module):
    def __init__(self):
        super(MyModel, self).__init__()
        self.a = nn.Parameter(torch.ones([1,1],dtype=torch.double), requires_grad=True)       ###
        self.b = nn.Parameter(torch.ones([1,1],dtype=torch.double)*torch.sqrt(torch.Tensor([4.]).double()), requires_grad=True)       ###
        self.register_parameter('aa',self.a)                                                                 ###
        self.register_parameter('bb',self.b) 
        
    def forward(self, x):
        x = x * self.aa * self.bb
        return x

model = MyModel()
print(model.a)
> Parameter containing:
tensor([[1.]], dtype=torch.float64, requires_grad=True)
print(model.b)
> Parameter containing:
tensor([[2.]], dtype=torch.float64, requires_grad=True)
print(model.aa)
> Parameter containing:
tensor([[1.]], dtype=torch.float64, requires_grad=True)
print(model.bb)
> Parameter containing:
tensor([[2.]], dtype=torch.float64, requires_grad=True)

As you see, you’ll get the expected values as well as types.

1 Like