While looking at vocab.txt here, I was left wondering why the vocabulary for Bertweet is not continuous.
For instance, see the below sample:
...
dice 63328
)@@ 63327
struggled 63326
wraps 63324
Investors 63312
#summer@@ 63305
...
As you can see, after 63305, we have 63312, followed by 63324… what about the numbers in between?
- Also, it feels a bit strange why vocabulary starts at around 3800.
르@@ 3800
utory 3798
...
Any explanations will be really appreciated.