How to concatenate multiple bi-lstm layers

nlp
1

I am working on a relation extraction task between two entities in a sentence.
For the model, I want to use Bi-LSTM model that takes three different parts of a sentence as a input:

  1. Left of the first entity 2. Right of the second entity 3. text between the two entities.

In Keras, it seems that you create a separate LSTM for each of the input and concatenate all three using Concatenate as below.

 left_ph = Input((None,), dtype="string")
 bet_ph = Input((None,), dtype="string")
 right_ph = Input((None,), dtype="string")
 left_embs = bilstm(left_ph, rnn_state_size, num_buckets, embed_dim)
 bet_embs = bilstm(bet_ph, rnn_state_size, num_buckets, embed_dim)
 right_embs = bilstm(right_ph, rnn_state_size, num_buckets, embed_dim)
 layer = Concatenate(1)([left_embs, bet_embs, right_embs])

Since i’m new to Pytorch, i was wondering if there’s an approach to utilize multiple sentences in Bi-LSTM model for text-classification purpose.

2

I have not done relation extraction so I don’t know what is the ‘default’ way of doing this and from your message I am not sure whether you are looking for the same method as in your Keras code snippet, so I am going to outline two methods:

  1. Creating separate BiLSTMs for each of 3 texts, and the concatenating it (to later put it into FFNN I suppose).
  2. Concatenating 3 texts before parsing it into the BiLSTM.

Option nr 1:

import torch
from torch import nn

left = torch.randn((1, 20)) # random input vector as an example input
entity = torch.randn((1, 10)) # random input vector as an example input
right = torch.randn((1, 20)) # random input vector as an example input

left_bilstm = nn.LSTM(input_size=20, hidden_size=16, num_layers=2, dropout=0.5, bidirectional=True)
entity_bilstm = nn.LSTM(input_size=10, hidden_size=16, num_layers=2, dropout=0.5, bidirectional=True)
right_bilstm = nn.LSTM(input_size=20, hidden_size=16, num_layers=2, dropout=0.5, bidirectional=True)

(left_bilstm_hidden_states, _) = left_bilstm(left.unsqueeze(0))
(entity_bilstm_hidden_states, _) = entity_bilstm(entity.unsqueeze(0))
(right_bilstm_hidden_states, _) = right_bilstm(right.unsqueeze(0))

concatenated_bilstm_hidden_states = torch.cat((left_bilstm_hidden_states, entity_bilstm_hidden_states,
                                               right_bilstm_hidden_states), dim=2)

Option nr 2:

import torch
from torch import nn

left = torch.randn((1, 20)) # random input vector as an example input
entity = torch.randn((1, 10)) # random input vector as an example input
right = torch.randn((1, 20)) # random input vector as an example input

concatenated_input= torch.cat((left, entity, right), dim=1).unsqueeze(0) # concatenating input

bilstm = nn.LSTM(input_size=50, hidden_size=32, num_layers=2, dropout=0.5, bidirectional=True)

(bilstm_hidden_states, _) = bilstm(concatenated_input) # getting the hidden states
1 Like