Different unit test results when using pip and conda to install torch+cuda


I have torch (1.10.1), torchvision (0.11.2) and CUDA (11.3) installed in two different virtual environments (python 3.7.10). For one environment I used pip, for the other I used conda following the installation guide.

# pip

pip install torch==1.10.1+cu113 torchvision==0.11.2+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

# conda

conda install pytorch torchvision cudatoolkit=11.3 -c pytorch

When running the exact same code for unit testing an InceptionResnetV1 classifier from facenet_pytorch (2.5.2) using GPU (Quadro RTX 5000), I get different checksums for the pip and conda based environments.

Checksum is the average confidence value across all faces detected in a test image. Checksums also differ when using other models, for example yolo_v5.

I use same np.random.seed and torch.manual_seed.

Running the unit test on different computers with the same type of gpu produces the same results.

I’m on a Microsoft stack.

What causes the differences?

Thank you in advance