The code is:
generate_sp_model('example.csv', vocab_size=20000, model_prefix='example', model_type='char')
But I get the following because my lines are 7000 characters long:
trainer_interface.cc(287) LOG(WARNING) Found too long line (7034 > 4192).
trainer_interface.cc(289) LOG(WARNING) Too long lines are skipped in the training.
trainer_interface.cc(290) LOG(WARNING) The maximum length can be changed with --max_sentence_length=<size> flag.
trainer_interface.cc(315) LOG(INFO) Loaded all 0 sentences
trainer_interface.cc(321) LOG(INFO) Skipped 12048 too long sentences.
I see that there may be some way to run this from command prompt(see the third WARNING above). But is there a way to pass the max_sentence_length flag into the above code to make this work as a python function?
Thanks!