I somehow accidentally deleted optimizer.zero_grad() which caused the problems.
optimizer.zero_grad()