# Bert additional pre-training **URL:** \ **Category:** nlp\ **Created:** [February 20, 2020, 8:26pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539 "2020-02-20T20:26:35Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [February 20, 2020, 8:26pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/1 "2020-02-20T20:26:35Z")
I would like to use transformers/hugging face library to further pretrain BERT. I found the masked LM/ pretrain model, and a usage [example](https://huggingface.co/transformers/model_doc/bert.html#bertforpretraining), but not a training example. In the original BERT repo I have this [explanation](https://github.com/google-research/bert#pre-training-with-bert), which is great, but I would like to use Pytorch. I’m not looking to finetune the model, just pretrain it further on the IMDB dataset, starting with an already trained model. Also, I assume that extracting BERT weights without the masked LM head and loading it into a regular BERT model should not be an issue. ---
**Author:** ![Jerry\_Chi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/jerry_chi/32/20800_2.png) [@Jerry\_Chi](https://discuss.pytorch.org/u/Jerry_Chi)\ **Post date:** [February 21, 2020, 1:32am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/2 "2020-02-21T01:32:30Z")
Did you figure out how to do this? ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [February 21, 2020, 2:15am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/3 "2020-02-21T02:15:31Z")
For now, I’m going with the original BERT repo in tf, than will try converting to torch. as explained [here](https://huggingface.co/transformers/converting_tensorflow_models.html). ---
**Author:** ![krishansubudhi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishansubudhi/32/20760_2.png) [@krishansubudhi](https://discuss.pytorch.org/u/krishansubudhi)\ **Post date:** [February 21, 2020, 6:45am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/4 "2020-02-21T06:45:55Z")
Use this repo > **[microsoft/AzureML-BERT](https://github.com/microsoft/AzureML-BERT)** > > End-to-end recipes for pre-training and fine-tuning BERT using Azure Machine Learning service - microsoft/AzureML-BERT This is slightly old but works. Let me know if you face any issues. ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [February 21, 2020, 10:40am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/5 "2020-02-21T10:40:32Z")
This introduces the overhead of using Azure. I was looking for an example using the pytorch-pretrained-bert or transformers, maybe something with the model zoo - all native to Pytorch and agnostic to infrastructure. Thanks though. ---
**Author:** ![krishansubudhi](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/krishansubudhi/32/20760_2.png) [@krishansubudhi](https://discuss.pytorch.org/u/krishansubudhi)\ **Post date:** [February 21, 2020, 2:59pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/6 "2020-02-21T14:59:44Z")
This uses huggungface Bert. If you can set the rank and world size explicitly, then you don’t need to use azure. ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [February 22, 2020, 7:54am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/7 "2020-02-22T07:54:31Z")
Using the original Google repo and converting to torch works. [EDIT] one caveat though, the new model is much heavier and needs more computational time and power to use. ---
**Author:** ![macwiatrak](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/macwiatrak/32/20773_2.png) [@macwiatrak](https://discuss.pytorch.org/u/macwiatrak)\ **Post date:** [February 24, 2020, 4:05pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/8 "2020-02-24T16:05:51Z")
This might not be exactly what you are looking for, but it might be easier to follow the huggingface interface for finetuning (very well documented), augment it for a given task and then just use the hidden states of the BERT model to get the embeddings. ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [February 24, 2020, 6:03pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/9 "2020-02-24T18:03:52Z")
I specifically wanted to do the additional pretraining. It is one if the steps used in some papers in order to adjust the distribution. The masked lm and next sentence classifier are available in huggingface, but I haven’t seen any training examples. ---
**Author:** ![macwiatrak](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/macwiatrak/32/20773_2.png) [@macwiatrak](https://discuss.pytorch.org/u/macwiatrak)\ **Post date:** [February 25, 2020, 12:14am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/10 "2020-02-25T00:14:16Z")
[This](https://github.com/huggingface/transformers/blob/master/examples/run_language_modeling.py) looks relevant. You can see there an example of LM task, you can reuse it/build on it and create your own LM task inside which you will initialize the weights of bert with a pretrained version and then train it with your own data. ---
**Author:** ![Tykat](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/c6cbf5/32.png) [@Tykat](https://discuss.pytorch.org/u/Tykat)\ **Post date:** [October 6, 2020, 11:38am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/11 "2020-10-06T11:38:05Z")
The link @macwiatrak provided is giving a 404 back. Could you please provide an alternative link to the same page if still available? ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [October 7, 2020, 5:48am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/12 "2020-10-07T05:48:28Z")
I actually ended up using the official repo for additional pretraining and exporting to PyTorch. ---
**Author:** ![Tykat](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/t/c6cbf5/32.png) [@Tykat](https://discuss.pytorch.org/u/Tykat)\ **Post date:** [October 7, 2020, 6:23pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/13 "2020-10-07T18:23:56Z")
Hi! Thank you for your answer! Could you please provide the repo of your code if it is publicly available? Thank you. ---
**Author:** ![spadel](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8797f3/32.png) [@spadel](https://discuss.pytorch.org/u/spadel)\ **Post date:** [November 16, 2020, 2:50pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/14 "2020-11-16T14:50:10Z")
@maria I would be interested too 🙂 ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [November 17, 2020, 2:50pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/15 "2020-11-17T14:50:16Z")
> **[google-research/bert](https://github.com/google-research/bert)** > > TensorFlow code and pre-trained models for BERT. Contribute to google-research/bert development by creating an account on GitHub. ---
**Author:** ![spadel](https://discuss.pytorch.org/letter_avatar_proxy/v4/letter/s/8797f3/32.png) [@spadel](https://discuss.pytorch.org/u/spadel)\ **Post date:** [November 18, 2020, 11:46am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/16 "2020-11-18T11:46:07Z")
But that’s just the official Tensorflow version, do you also have your PyTorch version available? ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [November 18, 2020, 12:19pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/17 "2020-11-18T12:19:09Z")
I used tensorflow and than converted that to PyTorch. The official implementation is very user friendly. That’s part of the reason I didn’t post the solution here 🙂. ---
**Author:** ![BramVanroy](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/bramvanroy/32/28628_2.png) [@BramVanroy](https://discuss.pytorch.org/u/BramVanroy)\ **Post date:** [November 18, 2020, 12:38pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/18 "2020-11-18T12:38:30Z")
You can use the scripts provided in the `transformers` library. These are well-tested and provide many useful options. For BERT in particular, you can have a look [here](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling). If you have additional questions, you can use [their dedicated forum](https://discuss.huggingface.co/). ---
**Author:** ![maria](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/maria/32/14253_2.png) [@maria](https://discuss.pytorch.org/u/maria)\ **Post date:** [November 18, 2020, 1:49pm UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/19 "2020-11-18T13:49:29Z")
Awesome! I’m glad to see examples (didn’t find any at the time - 6mo ago). ---
**Author:** ![m-nlp-q](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/m-nlp-q/32/35721_2.png) [@m-nlp-q](https://discuss.pytorch.org/u/m-nlp-q)\ **Post date:** [March 12, 2021, 10:04am UTC](https://discuss.pytorch.org/t/bert-additional-pre-training/70539/20 "2021-03-12T10:04:35Z")
Hello @maria , did you use the [scripts](https://github.com/huggingface/transformers/tree/master/examples/language-modeling#robertabertdistilbert-and-masked-language-modeling) of the `transformers` library? If you did, which one do you suggest, Hugging Face or the official repo of BERT? The latter includes using TensorFlow and then export to PyTorch. Ideally, I would like to use just PyTorch and no TensorFlow.