ValueError: Unexpected grad_sample type: <class 'NoneType'> when trying to train with differential privacy

Opacus
1

Hello,

I am using PyTorch 1.10.0+cu102 and Opacus 1.0.0 and would like to use differential privacy for the training of an image classifier. When using optimizer.step() during training, I get the following ValueError: Unexpected grad_sample type: <class ‘NoneType’>

Specifically, this is my Traceback:

Traceback (most recent call last):
  File "run.py", line 102, in <module>
    sc.run_learning()
  File "/gpfs/home/user/diffp_project/run2.py", line 82, in run_learning
    le.run_learning_and_testing_process()
  File "/gpfs/home/user/diffp_project/LearningEnvironment.py", line 165, in run_learning_and_testing_process
    client.federated_round_local_train(model=self.global_model)
  File "/gpfs/home/user/diffp_project/Client.py", line 208, in federated_round_local_train
    epoch_loss_train = self.local_train(model=self.model)
  File "/gpfs/home/user/diffp_project/Client.py", line 176, in local_train
    running_loss = BatchIterator(model=model, phase=phase, Data_loader=train_loader, criterion=criterion, optimizer=optimizer, device=self.device)
  File "/gpfs/home/user/diffp_project/batchiterator.py", line 58, in BatchIterator
    optimizer.step()
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 478, in step
    if self.pre_step():
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 459, in pre_step
    self.clip_and_accumulate()
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 366, in clip_and_accumulate
    g.view(len(g), -1).norm(2, dim=-1) for g in self.grad_samples
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 313, in grad_samples
    ret.append(_get_flat_grad_sample(p))
  File "/home/user/miniconda3/envs/env5/lib/python3.7/site-packages/opacus/optimizers/optimizer.py", line 186, in _get_flat_grad_sample
    raise ValueError(f"Unexpected grad_sample type: {type(p.grad_sample)}")
ValueError: Unexpected grad_sample type: <class 'NoneType'>

Does anybody know how to fix it?

Here is some more information about my code:

The model is a torchvision.models.densenet121(pretrained=True), the optimizer is a torch.optim.SGD(self.model.parameters(), lr=0.1, momentum=0), the loss criterion is nn.BCELoss().to(self.device).

I also run model = ModuleValidator.fix(model), and of course model, optimizer, train_loader = self.privacy_engine.make_private(…) before the training.

Thanks in advance!

2

Hi @general,

Would you be able to share with us a reproducible code so we can debug it? I believe another user has reported this here:

3

Hi, I have commented on GitHub with exemplary code. Thanks!

4

Answered on Github

1 Like