Hi, I would like to know if there is a way to modify the model architecture during training, say for example, the last fc layer of a network, after the model was wrapped by DDP. The reason for this is because my training loop works as starting off with some number of warm up iterations and then after these iterations I’d have to change the last fc layer with a different output size. The difficulty lies in the way of trying to reference the last fc layer, since after DDP it became a DDP object. I wonder if there is a tweak for this? Thanks.
@Chung-Hao_Ku There might be unintended consequences of modifying the model under DDP. The reason being that DDP builds buckets and state based on the model passed in. Now if the model changes (ex: parameters added/removed), this might break DDP where it might no longer synchronize gradients for the new parameters.
My suggestion would be to create a new DDP instance with the new model and then continue training.
Yes. I realised later on it’s not common practice to do this and instead found an alternative. Thanks for your detailed explanation.