Migrating from keras to pytorch

hi every body. please help me.
in just want to transfer a code from keras to pytorch
first q: is this two models are same? if not what is exact equivalence code in pytorch:)
so:
is this code…?

  question = layers.Input(shape=(None,), dtype='int32', name='question')
        answer = layers.Input(shape=(None,), dtype='int32', name='answer')

        embedding = layers.Embedding(
            mask_zero=True,
            input_dim=self.vocab_size,
            output_dim=self.embedding_size,
            weights=[self.w2v_weights],
            trainable=True
        )

        lstm_1 = layers.LSTM(units=80, return_sequences=True)
        lstm_2 = layers.LSTM(units=80, return_sequences=False)
        question_model = lstm_2(lstm_1(embedding(question)))
        answer_model = lstm_2(lstm_1(embedding(answer)))

        sim = layers.dot([question_model, answer_model], axes=1, normalize=True)
        #     sim = layers.Activation(activation='sigmoid')(sim)
        qa_similary_model = models.Model(
            inputs=[question, answer],
            outputs=[sim],
        )
        qa_similary_model.compile(
            loss='mean_squared_error', optimizer='nadam', metrics=['accuracy'])

        embedding_model = models.Model(
            inputs=[question],
            outputs=[question_model]
        )
        return qa_similary_model, embedding_model

is equivalent to this?

   class LSTM_Model(Model):
        """
        a class to define multiple models
        """

        def __init__(self, **kwargs):
            super().__init__(**kwargs)

        def forward(self, question, answer):
            question_embedding = self.embeddings(question)
            # print("question embedding shape:", question_embedding.shape)
            answer_embedding = self.embeddings(answer)
            # print("answer embedding shape:", answer_embedding.shape)
            q_output, (qhidden, qcell) = self.lstm(question_embedding)
            # print("q_output shape:", q_output.shape)
            # print("qhidden shape:", qhidden.shape)
            # print("qcell shape:", qcell.shape)
            a_output, (ahidden, acell) = self.lstm(answer_embedding)
            # print("a_output shape:", a_output.shape)
            # print("ahidden shape:", ahidden.shape)
            # print("acell shape:", acell.shape)
            # qa_similary = torch.mm(qhidden[-1], ahidden[-1])
            # qa_similary =torch.matmul((qhidden[-1]), torc.th(ahidden[-1]))
            qa_similary = torch.matmul((q_output[-1]), torch.t(a_output[-1]), )
            # print("qa_similary shape:", qa_similary.shape)
            return qa_similary, qhidden

i need help :frowning:

You didn’t define any modules in your PyTorch code, so it shouldn’t be executable (or you didn’t post this code snippet).
Also, in Keras you are using two LSTM layers and pass the output of the first one to the second, while you are only using one in PyTorch, so you might want to add the second one.

tnx for you answering,yes i forget to copy it here, all code’s model is here:

class Model(nn.Module):
    def __init__(self, **kwargs):
        super().__init__()

        self.embeddings = nn.Embedding(num_embeddings=kwargs["vocab_size"],
                                      embedding_dim=kwargs["embedding_dim"],
                                      padding_idx=kwargs["pad_idx"])
        self.embeddings.weight.requires_grad = True  # to not refine-tune

        if kwargs["model"] == "lstm":
            self.lstm = nn.LSTM(input_size=kwargs["embedding_dim"],  # input
                                hidden_size=kwargs["lstm_units"],  # output
                                num_layers=kwargs["lstm_layers"],
                                bidirectional=False,
                                batch_first=True)
        if kwargs["model"] == "BiLSTM":
            self.lstm = nn.LSTM(input_size=kwargs["embedding_dim"],  # input
                                hidden_size=kwargs["bilstm_units"],  # output
                                num_layers=kwargs["bilstm_layers"],
                                bidirectional=True,
                                batch_first=True)

        self.dropout = nn.Dropout(kwargs["dropout"])
        self.tanh = F.tanh
        self.dropout = nn.Dropout(kwargs["dropout"])

    def forward(self):
        pass


class LSTM_Model(Model):
    """
    a class to define multiple models
    """
    def __init__(self, **kwargs):
        super().__init__(**kwargs)

    def forward(self, question, answer):
        question_embedding = self.embeddings(question)
        # print("question embedding shape:", question_embedding.shape)
        answer_embedding = self.embeddings(answer)
        # print("answer embedding shape:", answer_embedding.shape)
        q_output, (qhidden, qcell) = self.lstm(question_embedding)
        print("q_output shape:", q_output.shape)
        # print("qhidden shape:", qhidden.shape)
        # print("qcell shape:", qcell.shape)
        a_output, (ahidden, acell) = self.lstm(answer_embedding)
        print("a_output shape:", a_output.shape)
        # print("ahidden shape:", ahidden.shape)
        # print("acell shape:", acell.shape)
        # qa_similary = torch.mm(qhidden[-1], ahidden[-1])
        # qa_similary =torch.matmul((qhidden[-1]), torc.th(ahidden[-1]))
        q_output = q_output[-1]
        q_output = q_output.squeeze()
        a_output = a_output[-1]
        a_output = a_output.squeeze()
        mm = torch.mul((q_output), (a_output))
        mm -= mm.min(1, keepdim=True)[0]
        mm /= mm.max(1, keepdim=True)[0]
        qa_similary =torch.mean(mm, dim=1)
        # print("qa_similary shape:", qa_similary.shape)
        return qa_similary, qhidden

    print("**************************MODEL DEFINE & CREATED!****************************")

would you please say it work exactly as it is in keras?

I don’t think it will work exactly as in Keras, as it seems you are either still missing the second nn.LSTM.
I’m not deeply familiar with Keras, but I thought there is a Bidirectional layer, in case you want to use a bidirectional LSTM.
However, in the current Keras snippet, the two LSTM modules are piped to each other.
Could you explain a bit, how the Keras model works and how the lstms are interacting with each other?

I did a question matching task in keras, it has two input for two similar question.
Lets named them q,q’:slight_smile:
I have two lstm that q and q’ are passing from both of them,in this way:
q----->embedding----->lstm1----->lstm2—>result:sum_a
q’----->embedding----->lstm1----->lstm2—>result:sum_b
Ok?
So then the models output should be two arg:

  1. the q’ result (sum_a)
  2. Dot product of sum_a and sum_b

So it is the explanation in LSTM term of keras…

And:
In second BiLSTM model we have just one bilstm layer ofter embedding

q----->embedding----->bilstm—>result:sum_a
q’----->embedding----->bilstm1—>result:sum_b

and again the models output should be two arg:

  1. the q’ result (sum_a)
  2. Dot product of sum_a and sum_b

This was an another BILSTM explanation,
can you help about this two in torch?

Thanks for the description.
Based on it, it seems you are using the first approach with two LSTMs in Keras, while the second one with the bidirectional LSTMs in PyTorch.
If that’s the case, then surely these architectures are not the same. :wink:

I think I’m not familiar enough with Keras and don’t see how these two LSTM layers can be interpreted as a bidirectional LSTM, but maybe Keras applies something under the hood, which I’m missing.

1 Like