Difference between tensor.new_empty(temp.size()).normal_() and torch.randn(temp.size())?

[4158ndfkvBHJ1 (a)]

The question is basically in the title. I am reading the SWAG implementation (swa_gaussian/swag.py at master · wjmaddox/swa_gaussian · GitHub), and the sample_fullrank function uses gaussian sampling at line 127. The sample vector is defined as:

  cov_mat_sqrt.new_empty((cov_mat_sqrt.size(0),), requires_grad=False).normal_()

It seems like a really convoluted way of making a tensor of size cov_mat_sqrt.size(0) filled with gaussian values.

First I don’t really understand the point of tensor.new_empty(shape), what’s the difference with torch.empty(shape) ? Why call this function on a specific tensor if the new empty tensor is not going to inherit it’s shape?

And if that’s just initializing an empty tensor, is there any functionality difference between this line and a simpler torch.randn(cov_mat_sqrt.size(0)) ?

I understand that there may be different ways of achieving the same result in PyTorch, I just want to make sure I am not missing any subtle functionality of the implementation used in this codebase.

Thanks in advance to anybody who can clear this up

[ptrblck]

@4158ndfkvBHJ1

First I don’t really understand the point of tensor.new_empty(shape), what’s the difference with torch.empty(shape) ?

The new_* operations create a tensor with the same dtype and device as the source tensor it was called on, so it might be a convenient approach if you want to avoid using device=src.device, dtype=src.dtype in torch.empty.

@4158ndfkvBHJ1

And if that’s just initializing an empty tensor, is there any functionality difference between this line and a simpler torch.randn(cov_mat_sqrt.size(0)) ?

Yes, potentially the dtype and device as torch.randn would use its default (float32 and cpu).