Hi, I have a dynamic graph that adds/removes layers after some epochs.
I realized that the optimizer is not aware of newly added layer.
Q. Is it safe to call
opt.add_param_group() after a few iterations of
opt.step() in runtime?
Q. How can I delete some parameters from opt? (To save the memory in runtime)
There was a recommentation to reconstruct the optimizer:
but I think it will lose optimizer states like momentum for existing parameters. (Cloning the optimizer would be the last resort)