Papers have shown that NLP models pretrained on any dataset (even not language, e.g.: music, codes, brackets, etc) can improve the performance on other datasets.
So is there any pretrained nn.Transformer parameters available? I would really appreciate it if anyone could provide one. Specifically, I’m doing a music-related task, so it would be super amazing to find a nn.Transformer model pretrained on music data.
I usually refer to the HuggingFace pre-trained models. You can refer here.
I had worked on a music generation project (not directly relying on pre-trained models though).
My guess is that pre-trained models (like BERT, Roberta) may not perform well on music-related tasks because they weren’t trained on music data (rather on more general datasets like Wiki, Book corpus). It is still worth a try though.
1 Like
Thanks, but it seems that the Hugging Face provides highly integrated models for language-related tasks only. It involves predefined and pretrained tokenizers and other sort of things. Could I possibly extract the parameters of the models?
Yes, these pre-trained models are trained for language tasks. So, if you have to use them, you’ll need to create your own vocabulary and tokenizer. A better idea may be to just train the transformer model yourself. Even if you go extract the pre-trained weights and load it into your model, I don’t think it will help a lot.
I remember I found some useful resources on the Google Magenta website.