Hi, I am getting this error
RuntimeError: Expected to mark a variable ready only once. This error is caused by one of the following reasons: 1) Use of a module parameter outside the `forward` function. Please make sure model parameters are not shared across multiple concurrent forward-backward passes. or try to use _set_static_graph() as a workaround if this module graph does not change during training loop.2) Reused parameters in multiple reentrant backward passes. For example, if you use multiple `checkpoint` functions to wrap the same part of your model, it would result in the same set of parameters been used by different reentrant backward passes multiple times, and hence marking a variable ready multiple times. DDP does not support such use cases in default. You can try to use _set_static_graph() as a workaround if your module graph does not change over iterations.
Parameter at index 290 with name _forward_module.reward_engine.model.module.scoring_head.weight has been marked as ready twice. This means that multiple autograd engine hooks have fired for this particular parameter during this iteration.
From first glance, I do not see where in the program I used the the parameter
reward_engine.model.module.scoring_head.weight twice, I’d like to know if there are ways to implement hooks so that whenever this parameter is used, it writes a log or start an interactive debug session with ipdb