Hi,
I have a problem with optimization.step(), my train code:
previous = model.head[12].weight
for i, (image, target) in enumerate(bar):
image, target = image.to(device), target.to(device)
b_size = image.size()[0]
output = model(image)
loss = criterion(output, target)
now = model.head[12].weight
print('='*10)
print(previous,now)
print(model.head[12].weight.grad)
optimizer.zero_grad()
loss.backward()
optimizer.step()
The twelfth layer is nn.Linear(32, 1)
The previous and now variables always return the same:
tensor([[ 0.0034, -0.2354, -0.1383, -0.2392, -0.5482, 0.4603, 0.1331, 0.3651,
-0.1942, -0.1649, 0.2016, 0.3514, 0.2886, -0.0520, 0.1155, -0.3433,
-0.3107, 0.1798, 0.0551, -0.0161, 0.5021, 0.1313, 0.3340, 0.2885,
0.3380, -0.1972, 0.0067, 0.4465, -0.0814, -0.3551, 0.1053, 0.2180]],
device='cuda:0', requires_grad=True)
Even that layer has the gradient value:
# Iter 1: None
# Iter 2:
tensor([[-0.4360, -0.2952, -0.6197, -0.0216, -1.0340, 0.7521, 0.2088, 0.7050,
-0.6158, 0.4012, 1.0144, -0.5273, 0.6514, 0.2796, -0.6339, -0.8657,
-0.2789, 0.2030, -0.0475, 0.4096, 0.9676, 0.3229, 0.5224, 0.1033,
0.5484, -0.3745, -0.1743, 0.3179, -0.5385, -0.2508, 0.3502, 0.7884]],
device='cuda:0')
# Iter 3:
tensor([[-0.5874, -0.7407, -0.6531, 0.2646, -1.6728, 0.4518, 0.1702, 1.0924,
-1.2420, 0.1817, 0.7247, 0.9656, 0.0686, 0.1332, 0.3749, -1.8020,
-2.0433, 0.4756, -0.8883, -0.4920, 1.8353, -0.5856, -0.2931, 0.8012,
1.2388, -0.1963, 0.3465, 1.1411, -0.7416, -0.5985, -0.1804, 0.0322]],
device='cuda:0')
Is it a bug, or I did something wrong?