The only information I could find about parallel.gather is from this parallelism tutorial. I think you’re right that might be what I want to be using, But I don’t think I understand how to use it correctly. I’ve tried a few different ways to rework the code but when I call nn.parallel.gather(model.module.conv.values.average, 0) it still is always returning all 0’s with 2GPUs. Could you please provide any additional help? The backword hook is already in the forward call so it should be on both devices. I tried initializing the array after dataparallel as well. and I tried the dataprallel subclass from that tutorial fixed with the info here