Torch.manual_seed() not working

1

Hi everyone,

I am trying to reproduce identical random numbers using torch.manual_seed() but it does not seem to be working. I tried the below method afte referring a few existing forum posts. Please let me know what is wrong.

import torch
from torch import nn, optim
import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"

# torch.manual_seed(42)
import random
random.seed(42)
np.random.seed(42)
torch.cuda.manual_seed(42)
torch.backends.cudnn.deterministic = True

model_linear = LinearRegression()
model_linear.to(device=device)
model_linear.state_dict()

The last line gives random values every single time to the weights and biases. Please if anyone can help me out.

Python version: 3.12.4
Torch version: 2.4.1

2

You are initializing the model on the CPU so add torch.manual_seed() which should also seed the device btw.

3

I uncommented torch.manual_seed(42) but still my model weights and biases are always giving different initial values.

4

You can check torch.manual_seed with torch.nrand like the MPS bug:

5

Could you post the code for LinearRegression?

6

Sure.

device = "cuda" if torch.cuda.is_available() else "cpu"

class LinearRegression(nn.Module):
    '''
        Class to define the neural network using Linear layers. Importing nn.Module is necessary whenever building any NN
    '''

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.layer1 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)
        self.layer2 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        self.forward1 = self.layer1(x)
        return self.layer2(self.forward1)
7

When manual_seed is working:

>>> from torch import Generator, randn
>>> g = Generator(device='cpu')
>>> g.manual_seed(42)
>>> randn(3, generator=g, device='cpu') # initially missing
tensor([0.3367, 0.1288, 0.2345])
>>> g.manual_seed(42)
>>> randn(3, generator=g, device='cpu')
tensor([0.3367, 0.1288, 0.2345])

Can you get a different result on your environment?

8

When I run it the first time, I get the same 3 values as you got using this code. When I run randn(3, generator=g, device=‘cpu’) multiple times, I get subsequently different values.

Although I don’t understand how I can fix the different values I am getting in my code.

9

I checked again. So torch.manual_seed(42) is giving me the same initial values when I create the model. What I do not understand is if I am changing my device to GPU, why doesn’t torch.cuda.manual_seed(42) work then? What can I do to make it work?

10

Could you provide detailed information of your environment?

  • OS, CPU, etc.
  • How to install PyTorch: conda, pip, source
11

OS: Windows 11 64-bit operating system
CPU: Intel(R) Core™ i5-8300H CPU @ 2.30GHz 2.30 GHz

For installing Pytorch, used the command listed here: “https://pytorch.org/”
Installed Conda from “Download Now | Anaconda” and it came with Python 3.12.4 and pip 24.0

12

Thanks for the model definition. Your code works as expected and returns the same random values after seeding the code:

python tmp.py
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

python tmp.py 
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

python tmp.py 
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

cat tmp.py 
import torch
import torch.nn as nn

class LinearRegression(nn.Module):
    '''
        Class to define the neural network using Linear layers. Importing nn.Module is necessary whenever building any NN
    '''

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.layer1 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)
        self.layer2 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        self.forward1 = self.layer1(x)
        return self.layer2(self.forward1)

torch.manual_seed(42)

device = "cuda" if torch.cuda.is_available() else "cpu"
model_linear = LinearRegression()
model_linear.to(device=device)
print(model_linear.state_dict())

As previously explained:

The initial random values are created on the CPU before you are moving the model to the GPU, so you need to seed the host, too.

13

Could you try to test manual_seed after downgrading PyTorch to 2.3.1?

# CUDA 11.8
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia
# CUDA 12.1
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia
# CPU Only
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 cpuonly -c pytorch
14

Thanks for the help! @ptrblck

QQ: If I wish to initialize the model on GPU directly, what changes can I make to my code?

15

You can pass the device argument to the layer initialization directly e.g. via: self.layer1 = nn.Linear(..., device=device).

16

Hi,
I am using torch GCN, and although I set all seed values, I am still getting random node feature generation from the model. Would you mind helping me understand where the issue is coming from?

17

Are you using the following model?
https://pytorch-geometric.readthedocs.io/en/latest/generated/torch_geometric.nn.models.GCN.html#torch-geometric-nn-models-gcn

18

Thanks a lot for your reply! yes, this is the model I am using. However, in my case, I figured the issue was due to not resetting the parameters in the right place.