Finally back to this after debugging other things. Apparently the .to() method made it run but it doesn’t get the same values which I imagine is because its not actually keeping the numbers the same. I’m using a registered buffer as well. Started a thread with all my current code, please take a look when you get a chance 
Multi GPU Hook not correctly filling buffer