[DataLoader Problem] dataloader with data in different dimension

1

When working on object detection problem, there will be different number of object in one picture. For example, when writing my customize dataset, the ground-truth for object location([x, y, w, h]) will have different number in first dimension
Ex:
First image has 10 object in image, location ground-truth=[10,4];
Second image has 34 object in image, location ground-truth=[34,4]).

This will cause the error in dataloader where torch.stack will encounter dimension mismatch problem.
Is there any solution for this?

2

You can either pad so that they all have the same dimension, or disable pytorch auto stacking: collate_fn = lambda i: i

3

Do u mean that I have to write my own collate_fn (disable auto stacking)?

4

use the one that I put there (or you can write your own too)

5

Thanks for ur help, lambda i:i really work!!!
Say the batch size = 8, by using this method I have to iterate these 8 data seperately and add the loss???
And also Is it be able to use lambda i:i with dataparallel?

6

I’m not sure how your loss function work (maybe you can split 4 objects into 4 items or something), I just manually combine the data into a batch when I do it (basically your own collate_fn function)

maybe somebody can answer that, I’ve never done dataparallel so I don’t know

7

How about the first question??
It makes no difference between the aforementioned(batch_size=8 and iterate separately and add loss) and batch_size=1
correct me if Ive got something wrong here, thanks :slight_smile: )