High PCIE traffic

autograd
1

I’m getting a very high PCIE bandwidth usage training a UNet-like network, in the viscinity of 50% of PCIe 3.0 x16 for no apperent reason. Training on a single GPU.

Did you ever experience this sort of problem and, maybe, know how to fix this? Or at least, how to check what’s causing this, debug.

2

Backwards pass seems to be much more PCIE-intensive - few times so - than forward pass.

3

Hi, can you elaborate on how you are profiling PCIe bandwidth during training?