I would like to implement an architecture similar to this:
Characterizing and Improving Stability in Neural Style Transfer, Gupta, A. and Johnson, J. and Alahi, A. and Fei-Fei, L.
It is a Recurrent Convolutional Neural Network. The light blue box is a simple convolutional neural network and the rest of structure makes the network recurrent. The authors use a sequence of 10 frames long that gets unfolded in 10 steps. The network gets fed with the current frame and the previous stylized frame (the frame generated on the previous step).
I have a working implementation of the feedforward architecture (the light blue box in the picture) and I would like to transform it in a Recurrent Convolutional Neural Network. Unfortunately I could not find much about the topic in the Pytorch community.
I have two questions in order to transform a Convolutional Neural Network in a RCNN:
How can I prepare the dataset of frames in order to feed them to the RCNN? Should I make a Dataset class that returns a sequence of frames?
How can I unfold this sequence of frames in order to use back propagation through time? I read the Pytorch documentation and saw that I can not use a RNN, LSTM, GRU layer in this particular case but I should write the recursion myself.
I would much appreciate if you have suggestions, pointers, tutorials, videos I can take a look at in order to understand this part.