Pytorch Equivalent of tensorflow code for tokenizing a sentence

nlp
1

Hi I am new to deep learning…

I am trying to convert some Tensorflow code into Pytorch for practice

can you please help me understand what will be pytorch equivalent for the following code block:

def tokenize(lang):

  lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(filters='')
  lang_tokenizer.fit_on_texts(lang)

  tensor = lang_tokenizer.texts_to_sequences(lang)

  tensor = tf.keras.preprocessing.sequence.pad_sequences(tensor,padding='post',maxlen=20,dtype='int32')

  return tensor, lang_tokenizer

the code i am trying to convert is the following: