I’ve heard that running on GPU can give nondeterministic results, but is this expected to happen on CPU?
At the beginning of the code, I’ve called
torch.manual_seed(1234). I’ve also seeded the numpy and python random number generators just to be safe. I have a snippet in my code (inside a loop):
loss = self.loss_function(out, y) dl_dold_state, dl_dtheta_direct = grad(loss, (state_vec, self.theta), retain_graph=True)
And I have verified that even when all input variables (loss, out, y, state_vec, self.theta) match their values from the last run of the code (at the same loop iteration), dl_dtheta_direct can output a slightly different value (the error is on the order of 1e-9).
I’m running on a laptop without a GPU, so this code is definitely running on CPU. It may not seem like a big deal, but I’m getting unexpected behaviour in my code and if there is some possiblility of a bug in pytorch’s grad operation it opens up the possibility that the bug is not in my code but in pytorch.
Anyone had a similar problem or know how to resolve it?