Is there a simple bert model in pytorch? Because i know the base model is almost based on a normal transformer so the source code could be to long. I just can’t seem to find it.
You can implement a simple bert model, but you would I have to reduce of some parameters (like number of heads) and also the dimensions of some layers like the embedding layer. You can use torch.nn.TransformerEncoderLayer for implementation. To undersdand the mechanics behind the module, you can watch torch.nn.TransformerEncoderLayer - Part 1 and the subsequent videos.