Load pretrained BERT without Huggingface's transformers

For educational purpose, I would like to use BERT embedding as input to solve the SQuAD Dataset. So far, converting BERT pretrained model to a pytorch model does not work (Issues 393, 1619, cannot post more than 2 links), and most tutorial I find online uses Huggingface’s transformer, which is kinda not my taste since they provide much more than what I need.

Is there a way to load BERT using https://pytorch.org/docs/stable/nn.html#torch.nn.Embedding.from_pretrained? In a broader sense, is there a way such that when another embedding comes up, we can import it to the model without having to depend on the others?