There seems to be multiple ways of initializing a tensor from a distribution, for example if we want to sample from a uniform distribution I found this can be done by:
Method 1
from torch.distributions.uniform import Uniform
U = Uniform(low=torch.tensor([0.0]), high=torch.tensor([1.0]))
samples = U.sample(sample_shape=(5,5)) # This shape turns out 5x5x1 (?)
Method 2
w = torch.empty(3, 5)
nn.init.uniform_(w, a=0, b=1)
Method 3
# The arguments seem to be uniform_(from, to) but writing from gives
# me an error because it is a keyword
w = torch.empty(3, 5).uniform_(0, to=2)
Perhaps there are more ways to do this that I’ve missed. I feel using the nn.init seems to make sense, but from https://pytorch.org/docs/stable/nn.init.html it seems to be quite lacking in comparison to https://pytorch.org/docs/stable/distributions.html.
Question:
Which method is recommended to use and is there a reason why we have multiple ways of doing it? All of these methods are also a little bit different than the way numpy does it which seems to be most intuitive to me: s = np.random.uniform(low=0, high=1, size=(1, 5))
My personal point of view is:
-
torch.distributions
have a lot of sampling methods and other utility functions (which can be used in e.g. a reinforcement learning setup), so that I wouldn’t use them to just initialize a tensor.
- I’m using
nn.init
only to initialize nn.Modules
via model.apply(init_fn)
.
- Method 3 is the way I would prefer to initialize a tensor.
Of course your preference might be different and I can also imagine that Method 2 could be preferred.
1 Like
Thank you, I really appreciate you taking the time. Your personal viewpoint is good enough for me, method 3 is now my go to Only issue I have is if I for example want to sample from a poisson distribution is there a way to do that? I guess I’m just having trouble finding the documentation for the distributions we can sample from
To create a tensor from a poisson distribution you could directly call torch.poisson
on the rates of your Poisson distribution via:
rates = torch.rand(4, 4) * 5 # rate parameter between 0 and 5
torch.poisson(rates)
> tensor([[9., 1., 3., 5.],
[8., 6., 6., 0.],
[0., 4., 5., 3.],
[2., 1., 4., 2.]])
Alternatively, distributions.poisson.Poisson(rates).sample()
would also work, and it really depends on your personal preference here.