Using all_reduce

1

whenever I run my code using torch I get this error:

~\AppData\Local\Continuum\Anaconda3\envs\pytorch CVdevKit\lib\site-packages\CVdevKit\core\dist_utils.py in average_gradients(model)
     24     for param in model.parameters():
     25         if param.requires_grad and not (param.grad is None):
---> 26             dist.all_reduce(param.grad.data)
     27 
     28 def broadcast_params(model):

AttributeError: module 'torch.distributed' has no attribute 'all_reduce'

Now I found out that all_reduce should be in distributed_c10d which is in the distibuted folder, so I tried running the following:

from torch.distributed import distributed_c10d

~\AppData\Local\Continuum\Anaconda3\envs\pytorch CVdevKit\lib\site-packages\torch\distributed\rendezvous.py in <module>
      5 
      6 import os
----> 7 from . import FileStore, TCPStore
      8 
      9 

ImportError: cannot import name 'FileStore' from 'torch.distributed' (C:\Users\s158979\AppData\Local\Continuum\Anaconda3\envs\pytorch CVdevKit\lib\site-packages\torch\distributed\__init__.py)

It seems there is a whole bunch missing from my enviroment? Does anyone know how I can solve this? I’ve been stuck for a few hours already.

2

Looks like torch.distributed is not enabled in your env. Can you run is_available to find out.

Given the path in log, it seems you are using Windows? PyTorch distributed is not available on Windows yet. Please comment here if you need this feature.