Intialising cls_token and bert sep_token in pytorch for multimodal classification

vision
1

Hi,

I hope your doing well, I have a question regarding initialising cls tokens and Bert separator tokens. While building a network, can I initialise both of them in the following way in PyTorch:

Cls_tok = sep_tok = nn.Parameter(torch.randn(1,1, embedding_size))

I’m unable to find any resources on these in PyTorch, could anyone please tell me whether it’s correct or not

Thank you