ValueError: Unexpected grad_sample type: <class 'NoneType'> when trying to train with differential privacy


I am using PyTorch 1.10.0+cu102 and Opacus 1.0.0 and would like to use differential privacy for the training of an image classifier. When using optimizer.step() during training, I get the following ValueError: Unexpected grad_sample type: <class ‘NoneType’>

Specifically, this is my Traceback:

Traceback (most recent call last):
  File "", line 102, in <module>
  File "/gpfs/home/user/diffp_project/", line 82, in run_learning
  File "/gpfs/home/user/diffp_project/", line 165, in run_learning_and_testing_process
  File "/gpfs/home/user/diffp_project/", line 208, in federated_round_local_train
    epoch_loss_train = self.local_train(model=self.model)
  File "/gpfs/home/user/diffp_project/", line 176, in local_train
    running_loss = BatchIterator(model=model, phase=phase, Data_loader=train_loader, criterion=criterion, optimizer=optimizer, device=self.device)
  File "/gpfs/home/user/diffp_project/", line 58, in BatchIterator
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/", line 478, in step
    if self.pre_step():
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/", line 459, in pre_step
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/", line 366, in clip_and_accumulate
    g.view(len(g), -1).norm(2, dim=-1) for g in self.grad_samples
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/", line 313, in grad_samples
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/", line 186, in _get_flat_grad_sample
    raise ValueError(f"Unexpected grad_sample type: {type(p.grad_sample)}")
ValueError: Unexpected grad_sample type: <class 'NoneType'>

Does anybody know how to fix it?

Here is some more information about my code:

The model is a torchvision.models.densenet121(pretrained=True), the optimizer is a torch.optim.SGD(self.model.parameters(), lr=0.1, momentum=0), the loss criterion is nn.BCELoss().to(self.device).

I also run model = ModuleValidator.fix(model), and of course model, optimizer, train_loader = self.privacy_engine.make_private(…) before the training.

Thanks in advance!

Hi @general,

Would you be able to share with us a reproducible code so we can debug it? I believe another user has reported this here:

Hi, I have commented on GitHub with exemplary code. Thanks!

Answered on Github

1 Like