Thanks for the reply!
I deleted parts of the network and only inputted 1 image and still it gave me the same error. I created a second question with the reduced code.
The error was appearing stachostically, even with seeding. I’m faily certain the code does not use the gradients before calling backward() the first time. (The results are deterministic if it does not crash)