The following is my code. If I remove
inplace=True in relu, I do not have error. However, if I leave it, I will have
RuntimeError: one of the variables needed for gradient computation has been modified by an inplace operation error
def __init__(self, inplanes, planes, use_prelu, use_se_block, anchor=None): super(BasicBlock, self).__init__() self.anchor = anchor self.conv1 = conv3x3(inplanes, planes, stride=1) nn.init.xavier_normal_(self.conv1.weight) self.conv2 = conv3x3(planes, planes) nn.init.normal_(self.conv2.weight, mean=0, std=0.01) self.use_prelu = use_prelu if self.use_prelu: self.prelu1 = nn.PReLU(planes) self.prelu2 = nn.PReLU(planes) else: self.relu = nn.ReLU(inplace=True) self.use_se_block = use_se_block if self.use_se_block: self.se = SEBlock(planes) def forward(self, x): if self.anchor is not None: x = self.anchor(x) residual = x x = self.conv1(x) if self.use_prelu: x = self.prelu1(x) else: x = self.relu(x) x = self.conv2(x) if self.use_prelu: x = self.prelu2(x) else: x = self.relu(x) if self.use_se_block: x = self.se(x) x += residual return x
I consulted the usage from https://github.com/pytorch/vision/blob/master/torchvision/models/resnet.py and cannot understand why there is no such error when used in resnet but I encounter the error in my code.
The usage in resnet.py is also weird. It is initialized with
inplace=True. However, it is used with
x = self.relu(x) in forward. Doesn’t this defeat the purpose of
inplace=True? Is it equivalent to just writing
self.relu(x) without reassignment?