After searching all around stackoverflow and PyTorch forum I found the solution here: Fill value to matrix based on index - #5 by ptrblck
Here it’s how it should be, I have no clue why…
tcls = torch.zeros_like(preds[..., 5:][obj], device=config.DEVICE)
tcls[torch.arange(tcls.size(0)), targets[..., 5][obj].long()] = targets[..., 5][obj].float()
lcls = self.BCE_cls(preds[..., 5:][obj], tcls) # BCE```