Your system installations of CUDA and cudnn won’t be used, if you install PyTorch binaries with these libs.
E.g. conda install pytorch torchvision cudatoolkit=10.1 -c pytorch will install CUDA10.1 and cudnn in your current conda environment.
What kind of error message are you seeing and which GPU are you using?