Hey! Pytorch is amazing and I’m trying to learn how to use it at the moment. I have gotten stuck at training a simple RNN to predict the next value in a time series with a single feature value per timestep. Here is my net: class Net(nn.Module): def __init__(self,hidden_dim = 128,num_layers=1):…

Thanks for the great example! I am now able to train the network. But I still have a question about detach() and retain_graph: The current prediction should depend on all previous timesteps. If I dont detach the hidden state and train the network with backward(retain_graph=True) will the updated p…

If you don’t detach the hidden states, then each backward will trace all the way to the beginning of that sequence and update parameter gradients along the way. The gradients will be more accurate (correct gradient wrt full sequence) than backward at each time step. In that case, I suggest you direc…

Thanks again for the quick reply! The reason I dont feed the whole sequence at the same time to the network is that I want to try this out as “online training” where I would recieve one timestep-value at a time. As the sequence of data I have to evaluate my model on is already complete, would I get…

Yes, it would be the same if you don’t do detach and do retain_graph :slight_smile: . In terms of the incorrect gradients, I was indeed talking about the detach option. Sorry for the confusion. Are your training data very very long sequences? If they were multiple sequences, I’d still try bwd on f…

I started this thread not understanding at all how RNNs work in pytorch, now everything is clear to me. Thank you a lot! The sequences are pretty short but I still want to do this the right way and send in the whole sequence instead as it makes for cleaner testing and evaluating. Im going to mark …

Hey Adrian, Would it be possible to see the version of your net after you’ve applied this detach trick? After reading this topic I’m still not sure how to use it properly…

It was a while since I worked on this so my memory isnt too fresh. From what I understood detach works like this: #In this example i have 2 different nets. The input is fed to the first net. #The output of the first net is fed to the second net. #I only want to train the second net in this example…

Thank you Adrian! I think I’ve got the idea now.

Hi, I have a related problem if you don’t mind… I want to process a sequence (tokens) with LSTM by hand, meaning that I’m going through the sequence with a for-loop instead of giving the whole sequence to LSTM and let it process all-at-once (let’s say). The reason for doing this is not important …

A couple questions before I can answer your concerns. Are you reusing the same charLSTM in both places? The Is charLSTM an RNN class or an RNNCell class? The torch.cat( [ word_embeddings[i,:,:], char_rep[i,:,:] ] ) call will concat along dim 0. Depending on the class of your charLSTM, this has dif…

[Solved] Training a simple RNN

autograd

OcramDiGallura (Marco) December 20, 2017, 9:06am 26

OK, thanks.

I indeed answered to your questions in a new thread: