I am feeding (frames,h,w,3) images in vgg19 from which the extracted features (frames,512) are fed in lstm layer. On keeping batch size = 1, maximum frames in a video are 300 and size is 224X224. I am getting out of memory error. How can I resolve this and do the task in memory efficient way?
If I understand you correctly, you try to train a LSTM network on extracted features using VGG. And you only want to train the LSTM model, not the VGG one.
Why don’t you extract the features of all the frames using VGG and store them in a file. And then train the LSTM model on the extracted features. This would not only be more memory efficient, but also much faster since you can skip the forward pass of the VGG model for training the LSTM model.