I am trying to train a BERT-base model for text classification and sentiment analysis. I managed to create the dataloader objects but when I iterate through them (either through the iter() function or for loops), I get this error, followed by my IPython console hanging (running on Spyder)
2021-01-26 14:10:36.370089: I tensorflow/core/platform/cpu_feature_guard.cc:142] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN)to use the following CPU instructions in performance-critical operations: AVX AVX2
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2021-01-26 14:10:36.528957: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
2021-01-26 14:10:36.780247: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
2021-01-26 14:10:36.805938: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
2021-01-26 14:10:40.337708: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
2021-01-26 14:10:40.456556: W tensorflow/core/framework/cpu_allocator_impl.cc:81] Allocation of 93763584 exceeds 10% of free system memory.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "C:\Users\mcnat\anaconda3\envs\ds\lib\multiprocessing\spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "C:\Users\mcnat\anaconda3\envs\ds\lib\multiprocessing\spawn.py", line 126, in _main
self = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'CommentsDataset' on <module '__main__' (built-in)>
Any ideas on how to deal with this error?
(I have been following this tutorial (https://curiousily.com/posts/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python/), with some modifications as I am using it with my own dataset for survey comments classification)