In the past, I have been using Theano where I could select the convolution strategy (see: Theano convolution strategies.). When I reimplement my model in PyTorch the memory usage is significantly higher, suggesting the convolution strategy is different.
In PyTorch, I can find the definitions in this github line.
My question is two-fold:
- Am I correct if I deduce that
IMPLICIT_PRECOMP_GEMM is the default strategy in PyTorch?
- Can I select the strategy manually (preferably per convolution) or can this only be done through the benchmarking function?
The default strategy is autodetermined by CuDNN’s strategy selector (it’s not always PRECOMP_GEMM and cudnn has some smart heuristics about it).
You can kick-in the CuDNN autotuner (that runs all the algorithms on your problem size and picks the best one) by adding this line somewhere;
torch.backends.cudnn.benchmark = True
See https://github.com/pytorch/examples/blob/master/imagenet/main.py#L111 for example.
We have not exposed specific algorithm selections at the python level.
Is there a way to determine which of these strategies is selected by the benchmark tool? We have tried that benchmark, but our application requires a lot of memory and in the first convolutional layer we already run out of memory. A similar implementation in Theano did not have this problem, which is why we want to figure out which strategy is used.
Where can we find how the benchmark algorithm works? Does it also take memory into account or just computational efficiency? Thanks for the effort.
the benchmark algorithm does take free memory into account.
The code of interest is located here: