Backward pass only uses ~50% GPU utilization?

Hello there!

The graph below visualizes my GPU utilization per timestamp as a progress bar. It is mostly 99%, but has some drops in between. Interestingly, those drops correlate consistently to the times of the “loss.backward()” call. Any idea why the backward pass might not use 100% GPU or how to improve it?

If helpful: The final loss is an L1-loss between two RGB images. One of those images is created using F.grid_sample() and some pinhole camera backprojection and projection functions.

Thanks in advance!