Torch.manual_seed() not working

1

Hi everyone,

I am trying to reproduce identical random numbers using torch.manual_seed() but it does not seem to be working. I tried the below method afte referring a few existing forum posts. Please let me know what is wrong.

import torch
from torch import nn, optim
import numpy as np

device = "cuda" if torch.cuda.is_available() else "cpu"

# torch.manual_seed(42)
import random
random.seed(42)
np.random.seed(42)
torch.cuda.manual_seed(42)
torch.backends.cudnn.deterministic = True

model_linear = LinearRegression()
model_linear.to(device=device)
model_linear.state_dict()

The last line gives random values every single time to the weights and biases. Please if anyone can help me out.

Python version: 3.12.4
Torch version: 2.4.1

2

You are initializing the model on the CPU so add torch.manual_seed() which should also seed the device btw.

3

I uncommented torch.manual_seed(42) but still my model weights and biases are always giving different initial values.

4

You can check torch.manual_seed with torch.nrand like the MPS bug:

5

Could you post the code for LinearRegression?

6

Sure.

device = "cuda" if torch.cuda.is_available() else "cpu"

class LinearRegression(nn.Module):
    '''
        Class to define the neural network using Linear layers. Importing nn.Module is necessary whenever building any NN
    '''

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.layer1 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)
        self.layer2 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        self.forward1 = self.layer1(x)
        return self.layer2(self.forward1)
7

When manual_seed is working:

>>> from torch import Generator, randn
>>> g = Generator(device='cpu')
>>> g.manual_seed(42)
>>> randn(3, generator=g, device='cpu') # initially missing
tensor([0.3367, 0.1288, 0.2345])
>>> g.manual_seed(42)
>>> randn(3, generator=g, device='cpu')
tensor([0.3367, 0.1288, 0.2345])

Can you get a different result on your environment?

8

When I run it the first time, I get the same 3 values as you got using this code. When I run randn(3, generator=g, device=‘cpu’) multiple times, I get subsequently different values.

Although I don’t understand how I can fix the different values I am getting in my code.

9

I checked again. So torch.manual_seed(42) is giving me the same initial values when I create the model. What I do not understand is if I am changing my device to GPU, why doesn’t torch.cuda.manual_seed(42) work then? What can I do to make it work?

10

Could you provide detailed information of your environment?

  • OS, CPU, etc.
  • How to install PyTorch: conda, pip, source
11

OS: Windows 11 64-bit operating system
CPU: Intel(R) Core™ i5-8300H CPU @ 2.30GHz 2.30 GHz

For installing Pytorch, used the command listed here: “https://pytorch.org/”
Installed Conda from “Download Now | Anaconda” and it came with Python 3.12.4 and pip 24.0

12

Thanks for the model definition. Your code works as expected and returns the same random values after seeding the code:

python tmp.py
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

python tmp.py 
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

python tmp.py 
OrderedDict([('layer1.weight', tensor([[0.7645]], device='cuda:0')), ('layer1.bias', tensor([0.8300], device='cuda:0')), ('layer2.weight', tensor([[-0.2343]], device='cuda:0')), ('layer2.bias', tensor([0.9186], device='cuda:0'))])

cat tmp.py 
import torch
import torch.nn as nn

class LinearRegression(nn.Module):
    '''
        Class to define the neural network using Linear layers. Importing nn.Module is necessary whenever building any NN
    '''

    def __init__(self, *args, **kwargs) -> None:
        super().__init__(*args, **kwargs)
        self.layer1 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)
        self.layer2 = nn.Linear(in_features=1, out_features=1, bias=True, dtype=torch.float32)

    def forward(self, x: torch.Tensor) -> torch.Tensor:
        self.forward1 = self.layer1(x)
        return self.layer2(self.forward1)

torch.manual_seed(42)

device = "cuda" if torch.cuda.is_available() else "cpu"
model_linear = LinearRegression()
model_linear.to(device=device)
print(model_linear.state_dict())

As previously explained:

The initial random values are created on the CPU before you are moving the model to the GPU, so you need to seed the host, too.

1 Like
13

Could you try to test manual_seed after downgrading PyTorch to 2.3.1?

# CUDA 11.8
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=11.8 -c pytorch -c nvidia
# CUDA 12.1
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 pytorch-cuda=12.1 -c pytorch -c nvidia
# CPU Only
conda install pytorch==2.3.1 torchvision==0.18.1 torchaudio==2.3.1 cpuonly -c pytorch
14

Thanks for the help! @ptrblck

QQ: If I wish to initialize the model on GPU directly, what changes can I make to my code?

15

You can pass the device argument to the layer initialization directly e.g. via: self.layer1 = nn.Linear(..., device=device).