when I loaded a pretrained model and try to continue the training.I found when model executes
optimizer.step() it cause error as following:
File "/home/f523/anaconda3/envs/rsy/lib/python3.6/site-packages/torch/optim/adam.py", line 110, in step p.addcdiv_(exp_avg, denom, value=-step_size) RuntimeError: output with shape [1, 256, 1, 1] doesn't match the broadcast shape [2, 256, 1, 1]
So I check the
p.addcdiv by using
However when breakpoint appears in the except case, I output the
denom. I find they getting same shape.
denom.shape Out: torch.Size([2, 256, 1, 1]) exp_avg.shape Out: torch.Size([2, 256, 1, 1])
p.addcdiv_ still get same error information.
Dose output is different from
I use two gpu to train it. And I have already used one gpu and two gpu to reloaded it and both failed.
So what should I do ?