This has been replied in How does one make sure that the parameters are update manually in pytorch using modules?