ValueError: Requested tokenizer basic_english, valid choices are a callable that takes a single string as input,

nlp
1

basic_english tokenizer error in PyTorch attention tutorial.

import torchtext
from torchtext.data.utils import get_tokenizer
TEXT = torchtext.data.Field(tokenize=get_tokenizer("basic_english"),
                            init_token='<sos>',
                            eos_token='<eos>',
                            lower=True)

google colab notebook not found

https://github.com/pytorch/tutorials/blob/gh-pages/_downloads/transformer_tutorial.ipynb

2

Feel free to open up an issue on (pytorch/tutorials](https://github.com/pytorch/tutorials/issues). Could you also post the error messages?

3

I raised an issue, along with error messages on github.

4

Can you send me the issue?

5
6

Thanks. I have pinged the maintainers for pytorch/tutorial repo. I run the script locally and it’s fine. So very likely, there is an issue with the setup.

7

try this:

!pip3 install torchtext==0.4.0
4 Likes
8

this works, in colab they have installed torchtext version 0.3.1