I tried in many ways but I could not get the solution. I am implementing GRU layer in a model in c.
I have extracted weights and biases. Can you give me the equations to compute the gates and outputs.
I am using hidden size = 32
input_size =32
num_layers = 1
batch_first = true
I followed the equations at https://pytorch.org/docs/stable/nn.html at GRU section.
My expected output is 151x32
Output is matched with just 1x32 rest 150x32 is not matching.
I am not able to know, what is called the time step in implementation time.
If possible suggest me any document about implementing GRU in testing mode.
GRU (forward) behaves the the same (and as in the documentation you linked) between training and testing. The time step is the first dimension in the 3d input and output seq_len x batch x features. Maybe considering the (unit, use unsqueeze) batch size explicitly can help you (just in case you accidentally fed a single-time-step, 151 batch size into the GRU when you wanted 151 time step, batch size 1.
Thanks for the quick reply, can you fix my issue one more step. I am giving my code which I stuck for few days, please correct me.
Input dimension is 1x151x32
k = crnn_model.rnn.weight_ih_l0
p = crnn_model.rnn.weight_hh_l0
m = crnn_model.rnn.bias_hh_l0
l = crnn_model.rnn.bias_ih_l0
As mentioned, gru layer has 1 layer, I am feeding zeros as state. And updating with the new state after
calculating output.
This is the code giving me wrong results as i mentioned. Please solve my issue to go ahead, I am stuck here for days
ht = np.zeros(32)
for j in range(151):
for i in range(32):
sum_r = np.sum(k[i]*input[0][j]) + l[i+0] + m[i+0] + (np.sum(p[i]))*ht[i]
r = sum_r
r = 1 / (1 + math.exp(-r))
sum_z = np.sum(k[i+32]*input[0][j]) + l[i+32] + m[i+32] + np.sum(p[i+32])*ht[i]
z = sum_z
z = 1 / (1 + math.exp(-z))
n = np.sum(k[i+64]*input[0][j]) + l[i+64] +(m[i+64]+np.sum(p[i+64])*ht[i])*r
n = math.tanh(n)
out = (1-z)*n + z*(ht[i])
ht_past[i] = out
ht = ht_past
I would suggest rewriting it in PyTorch first using torch.nn.functional (with linear, sigmoid, and tanh). If you have to, you can refine that afterwards.
np.sum seems wrong, as you want to do reduction over one axis only.
May be I expressed the question in a wrong way. It is not libtorch. just checking the outputs by loading the model and extraction weights in python itself.
I converted them into numpy, then only I am using np.add. I skipped those steps in post.
Is that procedure correct that I followed in the view of time steps. Taking 32 features at a time.
I tried in C++ as long, long double and float also. May be if 1st time step output is perfectly match without any precision loss, it works. But i am not able to know how to do that. Because every time steps ’ output is depend on previous time steps. Is it ?
