I have been able to fix the seed and make sure that the generate weights and biases and data in the dataloader are the same cross two different machines/hardware.
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.deterministic = True
but the final results are still different, and I think it comes from optimization part and cuda and cudnn (please let me know if I am wrong). BAsed on my observation, the weights and dataloader and initializations are all consistent.
I was wondering if there is a way to also fix that part or even if there is way to check that?