I have been surprised by the behavior of PyTorch. What I have done is initialize the bias of the output layer of a neural network using some data statistics. So considering, for instance:
mean=data.mean(0) #numpy array std = data.std(0) #numpy array #inside the model there is a sequential for the output layer. The model receives mean and std as arguments f=#here is the sequential f.bias=nn.Parameter(torch.from_numpy(numpy.log(std**2)))#initialize
I was surprised because if I put this as above, instead of std.copy(), the underlying std tensor changes when the bias changes in the optimizer updates. I thought that, in these cases, a new instance of the tensor is created (and same for numpy). As far as I know:
a=numpy.random.randn(1000,1) b=torch.from_numpy(a) a=10# this also changes b c=b**2 # a new instance in memory is created
Any suggestions? Thanks.