Recently, I meet the same problem. I set the random seed to ensure the same weight initialization and the same input data.
For example, I run my model 2 times. A the first times, I save the input data and the weight of the model and the output data to numpy array. Then I run my model again, and also save the above data. I compare the difference between them. Because I set the random seed, so the weight of them and the input data are the same. But it is really strange that the output of them are different. I run my model on GPU. I
Then, I run the model on cpu, I found that the output of them are the same.
So I wonder why there will be computing error when we use GPU, is this the bug of pytorch or something else?
Hey man,
There are some layers which have some random operations in them, maybe you should look into that, I had also a similar problem with the difference between GPU and CPU, and it happened to be some layer doing non deterministic operations.
While my model contain BN, convolutional layer, convolutional transpose layer, and dropout layer. I think the BN, convolutional transpose and convolutional transpose layer have the deterministic operation. Also because I set the random seed, so the dropout layer is still the same as the different experiment.
But now I have solved the problem by add
torch.backends.cudnn.deterministic = True
Even I use GPU, the outputs are the same!!!
But I still do not know why this operation work~
1 Like