i trained two models based on I3D from mmaction2 config , one for RGB dataset and the second for optical flow , i need to fuse the best models but i need flexibility to fuse them at any layer or final stage classifier , i need design class that take the pretarined model (pth) as base and creat new model ,that i can make choice in which layer i concatenate outputs to feed than one branch , final classifier .
my first is to fuse them before feeding final head as i3d , the second time i will freeze up layers and fuse outputs to train new weights and bias of the same architecture of i3d (inflated resnet50)