# BERT for multiple sentences

**URL:** <https://discuss.pytorch.org/t/bert-for-multiple-sentences/150004>\
**Category:** nlp\
**Created:** [April 25, 2022, 9:09am UTC](https://discuss.pytorch.org/t/bert-for-multiple-sentences/150004 "2022-04-25T09:09:05Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![sandeep1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sandeep1/32/12454_2.png) [@sandeep1](https://discuss.pytorch.org/u/sandeep1)\
**Post date:** [April 25, 2022, 9:09am UTC](https://discuss.pytorch.org/t/bert-for-multiple-sentences/150004/1 "2022-04-25T09:09:05Z")

</div>

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.

However, I have a question.

If I have 3 sentences, which are s1 and s2 and s3, and our fine-tuning task is the same.

In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]+ s3 [SEP].

```auto
from transformers import BertTokenizer

model_name='allenai/scibert_scivocab_uncased'
model_name = 'bert-base-uncased'

tokenizer= AutoTokenizer.from_pretrained(model_name)

```

tokenizer([[“hello i am going to”,“Rome”]])

```auto
Outputs correct ie:

```

```auto
{'input_ids': [102, 121, 1972, 705, 8303, 147, 758, 26241, 188, 9319, 24616, 422, 6474, 129, 9724, 650, 3978, 190, 9319, 24616, 13903, 372, 1942, 19226, 24648, 4931, 137, 3594, 17345, 190, 6666, 422, 334, 12605, 30113, 475, 506, 3437, 131, 633, 10945, 5458, 263, 17094, 205, 103]}

```

But, while tokenizing three sentences like this :-  
`tokenizer([["hello i am going","to","Rome"]])`  
It give the following error :-  
`TypeError: TextEncodeInput must be Union[TextInputSequence, Tuple[InputSequence, InputSequence]]`

How do we input such a sequences to the BERT and it is justified to do so?

Thanks for any help!

---

<div class="post-metadata">

**Author:** ![Matias\_Vasquez](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/32/47847_2.png) [@Matias\_Vasquez](https://discuss.pytorch.org/u/Matias_Vasquez)\
**Post date:** [April 25, 2022, 4:20pm UTC](https://discuss.pytorch.org/t/bert-for-multiple-sentences/150004/2 "2022-04-25T16:20:39Z")

</div>

Hi,

I tried to reproduce the error, but could not do it. With your example, I get the following (no error)

```python
# Input
tokenizer(["hello i am going","to","Rome"])

# Output
# {'input_ids': [[101, 7592, 1045, 2572, 2183, 102], [101, 2000, 102], [101, 4199, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0], [0, 0, 0], [0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1], [1, 1, 1]]}

```

---

<div class="post-metadata">

**Author:** ![sandeep1](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/sandeep1/32/12454_2.png) [@sandeep1](https://discuss.pytorch.org/u/sandeep1)\
**Post date:** [April 26, 2022, 6:46am UTC](https://discuss.pytorch.org/t/bert-for-multiple-sentences/150004/3 "2022-04-26T06:46:32Z")

</div>

Sorry, I missed one bracket. Edited the question.  
What i want is 1 Dimensional ids of multiple sentences provided by BERT

---

<div class="post-metadata">

**Author:** ![Matias\_Vasquez](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/matias_vasquez/32/47847_2.png) [@Matias\_Vasquez](https://discuss.pytorch.org/u/Matias_Vasquez)\
**Post date:** [April 26, 2022, 6:15pm UTC](https://discuss.pytorch.org/t/bert-for-multiple-sentences/150004/4 "2022-04-26T18:15:01Z")

</div>

tl:dr;

your approach is correct. Don´t think you need to change it.

> [@sandeep1](#):
>
> In one way, I add special tokens and the input looks like [CLS]+s1+[SEP] + s2 + [SEP]+ s3 [SEP].

> **Long answer**
>
> If you wanted each sentence to be a batch, then you would do it as in the first post (before editing).
> 
> [Here](https://huggingface.co/docs/transformers/preprocessing) is more information about batches and preprocessing for the tokenizer.
> 
> ```python
> tokenizer(["hello i am going","to","Rome"])
> 
> ```
> 
> However, as you mentioned before, you do **NOT** want this, but rather 1 Dimensional ids of multiple sentences.
> 
> If you go to the [BertTokenizer](https://huggingface.co/docs/transformers/main/en/model_doc/bert#transformers.BertTokenizer) and to the [PreTrainedTokenizer](https://huggingface.co/docs/transformers/main_classes/tokenizer#transformers.PreTrainedTokenizer) documentation and look for the ` __call__ ` method you will see which parameters it accepts.
> 
> The most relevant for us right now are `text` and `text_pair`. Both accept the same types described below.
> 
> ### `text` + `text_pair`
> 
> Both sentences are merged into a 1D vector, separated with the `[SEP] = 102` token.
> 
> ```python
> tokenizer(text="hello i am going, to", text_pair="Rome")
> # {'input_ids': [101, 7592, 1045, 2572, 2183, 1010, 2000, 102, 4199, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0, 1, 1], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1, 1, 1]}
> 
> ```
> 
> # Parameters accepted in text / text\_pair
> 
> ### str
> 
> This is the most simple case. You only input one `str`. This can be a word or a sentence.
> 
> ```python
> tokenizer("hello i am going to Rome")
> #{'input_ids': [101, 7592, 1045, 2572, 2183, 2000, 4199, 102], 'token_type_ids': [0, 0, 0, 0, 0, 0, 0, 0], 'attention_mask': [1, 1, 1, 1, 1, 1, 1, 1]}
> 
> ```
> 
> ### List[str]
> 
> Here we define a batch of sentences to be tokenized.
> 
> ```python
> tokenizer(["hello i am going","to","Rome"])
> # {'input_ids': [[101, 7592, 1045, 2572, 2183, 102], [101, 2000, 102], [101, 4199, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0], [0, 0, 0], [0, 0, 0]], 'attention_mask': [[1, 1, 1, 1, 1, 1], [1, 1, 1], [1, 1, 1]]}
> 
> ```
> 
> ### List[List[str]]
> 
> Now this is the case you are trying to do. However, even though it is not explicit anywhere (at least I did not find it), this list of lists is **not** for sequences of any length. As far as I understood, the outer list refers to the batches. However, the inner list is for a **pair** of sentences that will be treated like `text` - `text_pair`. If you enter more than 2 sentences in a batch, you will get an error.
> 
> Here is an example for two batches adding the special token manually.
> 
> ```python
> encoded_input = tokenizer(text=[["hello i am going [SEP] to [SEP] Rome", "other sentence"], ["next", "batch"]])
> 
> print("input: ", encoded_input)
> print("batch 1: ", tokenizer.decode(encoded_input["input_ids"][0]))
> print("batch 2: ", tokenizer.decode(encoded_input["input_ids"][1]))
> 
> ```
> 
> ```python
> # Output:
> # input: {'input_ids': [[101, 7592, 1045, 2572, 2183, 102, 2000, 102, 4199, 102, 2060, 6251, 102], [101, 2279, 102, 14108, 102]], 'token_type_ids': [[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1], [0, 0, 0, 1, 1]], 'attention_mask': [[1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1], [1, 1, 1, 1, 1]]}
> # batch 1: [CLS] hello i am going [SEP] to [SEP] rome [SEP] other sentence [SEP]
> # batch 2: [CLS] next [SEP] batch [SEP]
> 
> ```
> 
> As you can see, the manually added special tokens are treated the same when encoding and decoding the sentence. The problem is that `token_type_ids` can **only** be `0` or `1`, there are no more `ids` for a third sentence.
> 
> You can use a visualization tool like [`BertViz`](https://github.com/jessevig/bertviz) to see how your sentences behave and if it is your desired behavior.
> 
> - No ```[`SEP]`` token
> 
> `going` attends to `to`
> 
> ![image](https://discuss.pytorch.org/uploads/default/original/3X/0/b/0bf066e7ea5bc66f54794dba5b9cc44f0d2a1e7e.png)
> 
> - With `[`SEP]`` token `going`attends to`[SEP]`, almost does not notice `to```  
> ![image](https://discuss.pytorch.org/uploads/default/original/3X/0/7/0705870c45e9680b9047f9be410f7a4e4d257c51.png)

Hope this helps 😄