# Freezing layers issue for parallel GPU **URL:** \ **Category:** vision\ **Created:** [March 18, 2020, 4:17pm UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658 "2020-03-18T16:17:46Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![Han\_Brian\_Lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/han_brian_lee/32/14578_2.png) [@Han\_Brian\_Lee](https://discuss.pytorch.org/u/Han_Brian_Lee)\ **Post date:** [March 18, 2020, 4:17pm UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658/1 "2020-03-18T16:17:46Z")
Hi, So I think I found out what the cause for having weird error (see below) when trying to freeze layers using the following the following method (context manager also fails to freeze layers) is using DataParallel (model = nn.DataParallel(model)) across multiple GPUs. I’ve been running my model on 2 identical GPUs (gtx1080) and when I tried to freeze weights, I got the error shown below. When I don’t apply DataParallel and just use single GPU, freezing layers works like a charm as shown in countless examples across the internet and on this forum, but when DataParallel is used, it gives out a weird error (doesn’t make sense to me… the layers it should try to freeze are not leaf variables) Could anyone help me understand what is happening and **if this is a bug in pytorch?** (regardless of whether it is or not, I’d like to know **if there is a workaround** ) ```auto for param in model.parameters(): param.requires_grad = False ``` **RuntimeError: you can only change requires\_grad flags of leaf variables. If you want to use a computed variable in a subgraph that doesn’t require differentiation use var\_no\_grad = var.detach().** Thanks, ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 19, 2020, 2:46am UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658/2 "2020-03-19T02:46:36Z")
Could you post a code snippet to reproduce this issue? This dummy code snippet works: ```python model = models.resnet18() for param in model.parameters(): param.requires_grad = False model = nn.DataParallel(model) model.cuda() out = model(torch.randn(8, 3, 224, 224)) print(out.shape) ``` ---
**Author:** ![Han\_Brian\_Lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/han_brian_lee/32/14578_2.png) [@Han\_Brian\_Lee](https://discuss.pytorch.org/u/Han_Brian_Lee)\ **Post date:** [March 23, 2020, 5:02pm UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658/3 "2020-03-23T17:02:53Z")
Here it is. If you comment out the model = nn.DataParallel(model) line, no error occurs, otherwise the error occurs (on a computer with multiple GPUs): ```auto from __future__ import print_function from __future__ import division import torch import torch.nn as nn import torch.optim as optim import numpy as np class mynet(torch.nn.Module): def __init__ (self): super(mynet, self). __init__ () self.fc = nn.Linear(2, 1, bias=False) def forward(self, x): for param in self.fc.parameters(): param.requires_grad = False return self.fc(x) model = mynet() model = nn.DataParallel(model) # comment out this line to make the error go away model.cuda() data = np.array([0.0, 1.0]) x = torch.from_numpy(data).float().unsqueeze(0).unsqueeze(0).unsqueeze(0) gt = torch.from_numpy(np.array([0.0])).float().unsqueeze(0).unsqueeze(0).unsqueeze(0) # model.train() result = model(x=x.cuda()) ``` ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 24, 2020, 2:34am UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658/4 "2020-03-24T02:34:18Z")
Thanks for the code. The error is most likely thrown, as you are trying to manipulate the `require_grad` attribute of the model replica parameters. If you freeze the parameters outside of the `forward` method, it’ll work. What’s your workflow that you want to freeze them in each forward pass? ---
**Author:** ![Han\_Brian\_Lee](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/han_brian_lee/32/14578_2.png) [@Han\_Brian\_Lee](https://discuss.pytorch.org/u/Han_Brian_Lee)\ **Post date:** [March 24, 2020, 11:22pm UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658/5 "2020-03-24T23:22:26Z")
Hm… I see. I ideally wanted to train some iterations with a part of my network frozen, and some iterations after that with it not frozen, and alternate. I have 4 sub-networks(a,b,c,d). And there’s a big wrapper network (say N) that contains those 4 sub-networks, where the wrapper network’s flow goes (a,b separately)-\>combined into c-\>then d. And I wanted to freeze subnetwork a’s weights (load pickled trained weights for subnetwork a and don’t train them). 1. Not considering alternate freezing/unfreezing, what’s the best way to achieve freezing weights in this case? Would it be like the following? **But Once DataParallel wraps the model, the “a” subnetwork is no longer accessible from model.** What’s the best way of freezing subnetwork a’s weights for multi-GPU setting? Some way of maybe model.modules()? or model.\_modules[‘module’].a.parameters() ? ```auto model = N() model = nn.DataParallel(model) for parameter in model.a.parameters(): parameter.requires_grad = False ``` 1. If I wanted to alternate freezing/unfreezing, what’s the ideal way? ---
**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\ **Post date:** [March 24, 2020, 11:46pm UTC](https://discuss.pytorch.org/t/freezing-layers-issue-for-parallel-gpu/73658/6 "2020-03-24T23:46:12Z")
You should still be able to freeze the submodules via `model.module.a` after you’ve wrapped them in `nn.DataParallel`. As long as your changes are done outside of the forward pass, which is executed on each replica, it should be fine.