Fix random weights initialized by models in default

1

First of all, I know how to fix the randomness of the used weights if I set them manually for the model layers by using (torch.manual_seed(a number) )

my question,
when I create a model, it initializes the weights and biases by default using random values. How to fix the randomness here if I am using the default initialization for the weights and biases?
I need it to be able to reproduce the results.
thanks

2

Seeding the code should work as seen here:

def checksum(model):
    s = torch.sum(torch.stack([p.double().abs().sum() for p in model.parameters()]))
    return s

for _ in range(10):
    torch.manual_seed(0)
    model = models.resnet50()
    s = checksum(model)
    print(s)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764318.5419, dtype=torch.float64, grad_fn=<SumBackward0>)
    
for _ in range(10):
    torch.manual_seed(1)
    model = models.resnet50()
    s = checksum(model)
    print(s)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
# tensor(764287.9182, dtype=torch.float64, grad_fn=<SumBackward0>)
1 Like
3

Thanks for replying,
so basically it is the same, if I want to set the weights manually or keep the default ones …
One more question please,
what is the used approach for weight initialization, is it Xavier ?

4

I don’t understand this statement.
My code shows that seeding the code will result in the same parameters created by the default random init methods for each layer. Manually setting the weights will of course use your manually defined values.

It depends on the layer and is defined in the reset_parameters method.
E.g. nn.Linear uses this code:

    def reset_parameters(self) -> None:
        # Setting a=sqrt(5) in kaiming_uniform is the same as initializing with
        # uniform(-1/sqrt(in_features), 1/sqrt(in_features)). For details, see
        # https://github.com/pytorch/pytorch/issues/57109
        init.kaiming_uniform_(self.weight, a=math.sqrt(5))
        if self.bias is not None:
            fan_in, _ = init._calculate_fan_in_and_fan_out(self.weight)
            bound = 1 / math.sqrt(fan_in) if fan_in > 0 else 0
            init.uniform_(self.bias, -bound, bound)
1 Like
5

Sorry for inconvenience for the first part, I meant that (torch.manual_seed(a number) ) is used either with the default random initialization or with the manually set approach for random initialization e.g., change the initially defined weights to Xavier manually. I understood from your answer is YES we use the torch.manual_seed().

The last question please,
is there a way to make the model initializing all layers by default with Xavier?

6

Yes, you can use torch.nn.init.xavier_normal_/.xavier_uniform_ to initialize parameters by passing them directly to this method or by using it in a function passed to model.apply.