When using mix precison, i am getting this warning. Without mix precision, there is no such warning
UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()
scaler = torch.cuda.amp.GradScaler()
with experiment.train():
for batch_idx, _data in enumerate(train_loader):
image, labels= _data
image, labels = image.to(device), labels.to(device)
optimizer.zero_grad()
with torch.cuda.amp.autocast():
output = model(image)
loss = criterion(output, labels)
scaler.scale(loss).backward()
scaler.step(optimizer)
scaler.update()
scheduler.step()
This is how i defined the learning rate
scheduler = optim.lr_scheduler.OneCycleLR(optimizer, max_lr=hparams['learning_rate'],
steps_per_epoch=int(len(train_loader)),
epochs=hparams['epochs'],
anneal_strategy='linear')
Here is warning i am getting
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:1565: UserWarning: To get the last learning rate computed by the scheduler, please use `get_last_lr()`.
"please use `get_last_lr()`.", UserWarning)
/usr/local/lib/python3.7/dist-packages/torch/optim/lr_scheduler.py:134: UserWarning: Detected call of `lr_scheduler.step()` before `optimizer.step()`. In PyTorch 1.1.0 and later, you should call them in the opposite order: `optimizer.step()` before `lr_scheduler.step()`. Failure to do this will result in PyTorch skipping the first value of the learning rate scheduler