hadaev8
September 15, 2020, 12:26pm
1
Should anyone explain why here clone() used?
# # Use the max. for normalizing running avg. of gradient
# denom = max_exp_avg_sq.sqrt().add_(group['eps'])
# else:
denom = exp_avg_sq.sqrt().add_(group['eps'])
bias_correction1 = 1 - beta1 ** state['step']
bias_correction2 = 1 - beta2 ** state['step']
step_size = group['lr'] * math.sqrt(bias_correction2) / bias_correction1
##dropout learning rate
lr_dropout = step_size / denom.clone()
lr_dropout=lr_dropout * mask
p.data.addcmul_(-1, exp_avg, lr_dropout)
# p.data.addcdiv_(-step_size, exp_avg2, denom)
return loss
Whats general rule about it?
As far as I understand, this command copies the variable, in case I later modify it.
ptrblck
September 17, 2020, 8:10am
2
The clone() might not be necessary.