Why RNN needs two biases?

1

In the RNN implementation, there are two biases, b_ih and b_hh.
Why is this? Is it different from just use one bias?
Will it affect performance or efficiency?

2 Likes
2

You mean in general? For the same reason that it needs two sets of weights, one for the input and one from the previous state.

3

Taking RNN with tanh activation for example, it follows
h_t = tanh(w_{ih} * x_t + b_{ih} + w_{hh} * h_{(t-1)} + b_{hh})
b_{ih} and b_{hh} are just biases (trainable constants). It does not matter for input or previous state.
Actually, let b=b_{ih} + b_{hh},
h_t = tanh(w_{ih} * x_t + w_{hh} * h_{(t-1)} + b).
It should be the same.

1 Like
4

As you pointed out it doesn’t really change the definition of the model, but this is what cuDNN does, so we’ve made our RNNs consistent with this behaviour.

1 Like
5

OKay. Good to know. Thanks.