TL;DR
I got 30% token prediction accuracy for pretraining GPT2. Is it normal accuracy?
The Loss was about 4.2, The PPL was about 19
(Tools: Huggingface GPT2, ByteBPE, Deepspeed)
This is my report for pre-training gpt2 with conversational sentence
Because of short utterance, I only trained for short ‘nctx’. This is my configuration for gpt2
# 24 layer, 16 head, 1024 embed,
kogpt2_config_345m = {
"initializer_range": 0.02,
"layer_norm_epsilon": 1e-05,
"n_ctx": 64,
"n_embd": 1024,
"n_head": 16,
"n_layer": 24,
"n_positions": 64,
"vocab_size": 32000,
"activation_function": "gelu"
}
Also, the vocab is huggingface byte-level BPE and the model is huggingface GPT2LMHeadModel
And I trained the model with RTX8000 * 2 (which are 48GB*2)
Also, I used deepspeed with thisconfig
{
"train_batch_size": 1792,
"gradient_accumulation_steps": 2,
"optimizer": {
"type": "Adam",
"params": {
"lr": 1e-3
}
},
"fp16": {
"enabled": true,
"loss_scale": 0
},
"zero_optimization": {
"stage": 2,
"allgather_partitions": true,
"allgather_bucket_size": 5e8,
"overlap_comm": false,
"reduce_scatter": true,
"reduce_bucket_size": 5e8,
"contiguous_gradients": false,
"cpu_offload": true
},
"scheduler": {
"type": "WarmupLR",
"params": {
"warmup_min_lr": 0,
"warmup_max_lr": 1e-3,
"warmup_num_steps": 4000
}
}
}
And my loss is
outputs = self.model_engine(token_ids) # (batch_size, max_len, embed_dims)
logits = outputs.logits
# Cross Entropy
loss = self.loss_function(logits.transpose(2, 1), label) # (batch_size, max_len)
mask = mask.half()
loss = loss.half()
...
const = torch.zeros(1).to(args.local_rank).half()
masked_loss = torch.where(mask == 1, loss, const)
# Max integer of fp16 is 65536.0
# The sum of loss should not be larger than the max integer
# Thus, the loss is averaged for sequence length first and those values are averaged on batch size
sub_loss = masked_loss.sum(dim=-1)
sub_mask = mask.sum(dim=-1)
sub_avg = sub_loss/sub_mask
loss_avg = sub_avg.mean()
self.model_engine.backward(loss_avg)
Because of fp16 limitations, I used mean of mean to get global mean of loss batch
And finally, I got this result (Loss (eval) ~= 4.2, Acc (eval) ~= 30%, PPL ~= 19 )
Is it normal to get 30% accuracy on prediction token?
The PPL is similar with that of the GPT2, which is known as 17
Thank you.