How to initialize tensors from different distributions

There seems to be multiple ways of initializing a tensor from a distribution, for example if we want to sample from a uniform distribution I found this can be done by:

Method 1

from torch.distributions.uniform import Uniform
U = Uniform(low=torch.tensor([0.0]), high=torch.tensor([1.0]))
samples = U.sample(sample_shape=(5,5)) # This shape turns out 5x5x1 (?)

Method 2

w = torch.empty(3,  5)
nn.init.uniform_(w, a=0, b=1)

Method 3

# The arguments seem to be uniform_(from, to) but writing from gives
# me an error because it is a keyword
w = torch.empty(3, 5).uniform_(0, to=2)

Perhaps there are more ways to do this that I’ve missed. I feel using the nn.init seems to make sense, but from https://pytorch.org/docs/stable/nn.init.html it seems to be quite lacking in comparison to https://pytorch.org/docs/stable/distributions.html.

Question:

Which method is recommended to use and is there a reason why we have multiple ways of doing it? All of these methods are also a little bit different than the way numpy does it which seems to be most intuitive to me: s = np.random.uniform(low=0, high=1, size=(1, 5))

My personal point of view is:

  • torch.distributions have a lot of sampling methods and other utility functions (which can be used in e.g. a reinforcement learning setup), so that I wouldn’t use them to just initialize a tensor.
  • I’m using nn.init only to initialize nn.Modules via model.apply(init_fn).
  • Method 3 is the way I would prefer to initialize a tensor.

Of course your preference might be different and I can also imagine that Method 2 could be preferred. :wink:

1 Like

Thank you, I really appreciate you taking the time. Your personal viewpoint is good enough for me, method 3 is now my go to :slight_smile: Only issue I have is if I for example want to sample from a poisson distribution is there a way to do that? I guess I’m just having trouble finding the documentation for the distributions we can sample from

To create a tensor from a poisson distribution you could directly call torch.poisson on the rates of your Poisson distribution via:

rates = torch.rand(4, 4) * 5  # rate parameter between 0 and 5
torch.poisson(rates)
> tensor([[9., 1., 3., 5.],
          [8., 6., 6., 0.],
          [0., 4., 5., 3.],
          [2., 1., 4., 2.]])

Alternatively, distributions.poisson.Poisson(rates).sample() would also work, and it really depends on your personal preference here. :wink: