How to change the shape inside DataLoader?

vision
1

When training on a classification problem with multiclass and using CrossEntropyLoss, the targets should integer, and hence targets.long() is needed. But when training on multilabel or binary, we may invoke BCEWithLogitsLoss, which expects your targets to be in float.

I made changes in Dataset as follows:

if CRITERION_PARAMS.train_criterion_name == "BCEWithLogitsLoss":
    # Make changes to reshape rather than in Trainer.
    y = torch.as_tensor(y, dtype=torch.float32).view(-1, 1)
else:
    y = torch.as_tensor(y, dtype=torch.long)

However, I soon realized that this does not work since DataLoader will (collate?) the shape wrongly. What I expect is the following:

original_y_shape -> (batch_size, ) or (batch_size, 1)
modified_y_shape -> (batch_size, 1) # to match shapes with BCEWithLogitsLoss

But if I change the shape inside Dataset, the DataLoader gives me (batch_size, 1, 1) instead.

One easy way is to change the shape in the training function directly. But I would like to keep the changes in Dataset unless there is a strong reason not to (speed issues).

2

Starting torch 1.10 you can pass probabilities for each class into CrossEntropyLoss.
Also getitem method in custom dataset should return single element, not the batch.

3

Yes, getitem returns a single element, but when I reshape the single tensor with .view(-1, 1), the DataLoader somehow put an extra dimension 1 behind.

4

Right, so, for example, your self.labels inside dataset have shape (10000,1)
Then getitem will return single item (1), then you’re applying view(-1, 1), which transforms item to shape (1,1), then batch loader will return batch_num of elements with shape (1, 1)

5

So I actually do not need to use view(-1, 1) to ensure that the shape is correct to be passed in to BCEWithLogitsLoss?

6

Or you can do flatten() instead, then both (x,1) and (x) will result in (x)

7

Yes, I read up the documentation, you are right:

    By definition of collate function in DataLoader, it is mentioned in documentation that it prepends an extra dimension to the tensor as batch_size. Thus,
    if the input y is a tensor of shape [1,], then .view(-1, 1) will change the shape to [1, 1], and when we collate using DataLoader, say with batch_size = 4,
    then the collated y will be a tensor of shape [4, 1, 1] instead of [4, 1] since it prepends an extra dimension.