High PCIE traffic

I’m getting a very high PCIE bandwidth usage training a UNet-like network, in the viscinity of 50% of PCIe 3.0 x16 for no apperent reason. Training on a single GPU.

Did you ever experience this sort of problem and, maybe, know how to fix this? Or at least, how to check what’s causing this, debug.

Backwards pass seems to be much more PCIE-intensive - few times so - than forward pass.

Hi, can you elaborate on how you are profiling PCIe bandwidth during training?