Tracking down a suspected memory leak

This happens for me when using a CNN (no LSTM). Memory Usage/Leak

torch.backends.cudnn.enabled = False

does solve the problem for me, but unfortunately slows things down quite a bit.