# Custom loss for huggingface Trainer for sequences

**URL:** <https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221>\
**Category:** Uncategorized\
**Created:** [January 27, 2023, 6:35pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221 "2023-01-27T18:35:45Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [January 27, 2023, 6:35pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/1 "2023-01-27T18:35:45Z")

</div>

I have a dilemma, for the following custom loss I got this error:  
code:

```
class CustomTrainer(Trainer):
  def compute_loss(self, model, inputs, return_outputs=False):
    use_cuda = torch.cuda.is_available()
    device = torch.device("cuda" if use_cuda else "cpu")
    
    # forward pass
    outputs = model(**inputs)
    logits = outputs.get("logits")
   

    # logits: (batch_size, sequence_length, num_classes), reshape it to (batch_size*sequence_length, num_classes)
    predictions = logits.view(-1, model.config.num_labels)
    # shape: (batch_size*sequence_length)
    predictions = predictions.argmax(dim=-1)

    labels = inputs.get("labels").to(device)
    labels = labels.view(-1) 

    labels = torch.tensor(labels, dtype=torch.float)
    predictions = torch.tensor(predictions, dtype=torch.float, requires_grad=True)

    loss = loss_fct(predictions, labels)

    return (loss, outputs) if return_outputs else loss

```

Error:

> IndexError: Target 17 is out of bounds.

When I remove these lines of converting dtype:

```
#labels = torch.tensor(labels, dtype=torch.float)
#predictions = torch.tensor(predictions, dtype=torch.float, requires_grad=True)

```

I got this error:

> RuntimeError: Expected floating point type for target with class  
> probabilities, got Long

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [January 27, 2023, 6:40pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/2 "2023-01-27T18:40:58Z")

</div>

I guess you might be using `nn.CrossEntropyLoss` as the `loss_fct`?  
If so, note that this criterion accepts model outputs in the shape `[batch_size, nb_classes, *]` and targets as `LongTensor`s in the shape `[batch_size, *]` containing class indices in the range `[0, nb_classes-1]` as well as `FloatTensor`s in the same shape as the model output containing probabilities.

Also, these lines looks wrong:

```python
predictions = predictions.argmax(dim=-1)
predictions = torch.tensor(predictions, dtype=torch.float, requires_grad=True)

```

since you are detaching the `predictions` tensor from the computation graph in the `argmax` operation, which is not differentiable, and it seems you are trying to fix this by re-wrapping the tensor, which will not “re-attach” it to the computation graph.

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [January 28, 2023, 10:33am UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/3 "2023-01-28T10:33:25Z")

</div>

How to fix that?

I remove this line  
`predictions = predictions.argmax(dim=-1)`

and got this error:

> /usr/local/lib/python3.8/dist-packages/torch/nn/functional.py in cross\_entropy(input, target, weight, size\_average, ignore\_index, reduce, reduction, label\_smoothing)  
> IndexError: Target 17 is out of bounds.

the shapes as the following:

```auto
batch_size = 4
seq_length = 128
n_classes = 17

```

Before logits shape : `torch.Size([4, 128, 17])`

After predictions shape : `torch.Size([512, 17])`

Before labels shape : `torch.Size([4, 128])`  
After labels shape :` torch.Size([512])`

So the input of the CrossEntropyLoss:  
predictions shape : `torch.Size([512, 17]) : (batch_size * seq_length, n_classes)`  
labels shape : `torch.Size([512]): (batch_size * seq_length,)`

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [January 28, 2023, 4:26pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/4 "2023-01-28T16:26:49Z")

</div>

@ptrblck as you mentioned the required shape for the `CrossEntropyLoss`

input, I did the following and still get this error:

> IndexError: Target 17 is out of bounds.

```auto
predictions =logits.view(batch_size, model.config.num_labels, sequence_length) # shape: (batch_size, n_classes, seq_length)
labels = inputs.get("labels").to(device) # (batch_size, seq_length)

```

---

<div class="post-metadata">

**Author:** ![InnovArul](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/32/5282_2.png) [@InnovArul](https://discuss.pytorch.org/u/InnovArul)\
**Post date:** [January 28, 2023, 4:59pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/5 "2023-01-28T16:59:48Z")

</div>

Maybe you are already aware of this.  
The class labels should be from `0` to `n-1`. It seems you are using class labels from 1 to n?

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [January 28, 2023, 5:15pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/6 "2023-01-28T17:15:00Z")

</div>

I checked the dictionary `model.config.label2id` I used for both prepare labels for training, and the classes are from `[0-n]`: `[0-16]`.

---

<div class="post-metadata">

**Author:** ![InnovArul](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/innovarul/32/5282_2.png) [@InnovArul](https://discuss.pytorch.org/u/InnovArul)\
**Post date:** [January 28, 2023, 5:20pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/7 "2023-01-28T17:20:43Z")

</div>

> [@h.l.99](#):
>
> IndexError: Target 17 is out of bounds.

You may have to check where the indexing value `17` comes from.

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [January 28, 2023, 5:34pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/8 "2023-01-28T17:34:20Z")

</div>

I used the same training data with the default `Trainer` and it worked without causing this error. I think the default loss is `CrossEntropyLoss`, so it supposes not to have the target out of bounds.

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [January 28, 2023, 5:42pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/9 "2023-01-28T17:42:41Z")

</div>

Also I print the `labels` inside the `compute_loss`, and I didn’t see value 17 before the error appear.

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [February 5, 2023, 4:05pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/10 "2023-02-05T16:05:31Z")

</div>

@ptrblck can we do the same functionality of `predictions = predictions.argmax(dim=-1)` without detaching the `predictions` tensor from the computation graph?

---

<div class="post-metadata">

**Author:** ![h.l.99](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/h.l.99/32/56770_2.png) [@h.l.99](https://discuss.pytorch.org/u/h.l.99)\
**Post date:** [February 5, 2023, 4:25pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/11 "2023-02-05T16:25:23Z")

</div>

I have encountered the error of

> RuntimeError: element 0 of tensors does not require grad and does not have a grad\_fn

if I replace `argmax` with `torch.max` it shouldn’t detach the tensor from the graph.  
`_, predictions = torch.max(logits, dim=2)` or even this  
`_, predictions = torch.max(torch.tensor(logits, requires_grad=True), dim=2)`

Any idea? @ptrblck@InnovArul

---

<div class="post-metadata">

**Author:** ![ptrblck](https://discuss.pytorch.org/user_avatar/discuss.pytorch.org/ptrblck/32/1823_2.png) [@ptrblck](https://discuss.pytorch.org/u/ptrblck)\
**Post date:** [February 5, 2023, 8:32pm UTC](https://discuss.pytorch.org/t/custom-loss-for-huggingface-trainer-for-sequences/171221/12 "2023-02-05T20:32:02Z")

</div>

No, `argmax` is not differentiable as already mentioned.  
`torch.max` will return the max values (which are still attached to the computation graph) and the argmax (which will be detached as it’s not differentiable).
