Can I implement a Simple RNN this way? I couldn’t find any resources over this on the internet. For example, if I want to try a variant of LSTM or GRU could I define a class that has the new cell’s implementation and then just repeat it over time steps (i.e. sequence values in text).
Can I define recurrence over an arbitrary layer. Say, I want recurrence over a Linear layer. Could I just take the layer and implement recurrence over it by just looping through the layer outputs and applying the same layer again?
I don’t think there is another way to achieve the same.
You have to execute this sequentially since each iteration requires the result of the previous one.
The overhead of the loop should be neglectible, since GPUs work asynchronous and the loop only puts the kernel calls in a queue.