Sorry for misremembering it.
In this comment, he just recommended calling optimizer.zero_grad() before .backward().
optimizer.zero_grad()
.backward()
