Hi Everyone ,
My pytorch is version 0.2 compiled with cudnn6.0.21. When there is batchnorm in the neural net, batch_size can’t exceed 140000, otherwise it says:
"RuntimeError: CUDNN_STATUS_NOT_SUPPORTED. This error may appear if you passed in a non-contiguous input."
Without using batchnorm, batch_size can be very large unless breaking GPU memory. Does this mean that batchnorm doesn’t work with too large batch_size? Or is it a technical bug?