I have a variable length input for a neural network (e.g., CNN or MLP). The shape of this tensor is like [b, n, d], where b is the batch size, [n, d] is the feature for a sample, but only part of the data is valid. e.g., for n=3, d=1, a tensor may like [[1], [0], [0]] or [[1], [1], [0]] or [[1], [1], [1]], 0 means the padding.
How can I process this with CNN or MLP (I can flatten the tensor to 1D). I think I need some operation like masking?
You may use the Adaptive Pooling layer, which outputs the tensor of the same size regardless of the input size.
https://pytorch.org/docs/stable/generated/torch.nn.AdaptiveAvgPool2d.html
Yeah, but it seems the adaptive pooling does not support batch processing if the input size not equal?
Then, you can make a custom dataset using PyTorch and add adaptive pooling in the getitem function. Datasets & DataLoaders — PyTorch Tutorials 1.11.0+cu102 documentation. Then you can choose whatever batch size you want using the PyTorch data loader class.
Thank you for your suggestions. Another question is what are the advantages of this method compared to direct filling with 0?
So, one difference between these is, pooling will always reduce dimensions, and filling with 0s to make all input sizes equal will increase the dimensions.
So if we are adding a pooling layer in the getitem function, it will have the same general advantages that we get after adding a pooling layer inside the model (eg: reducing no. of training parameters, making the model more robust, etc), whereas adding 0s will not have any such effect.
Thank you for your suggestion. I’ll try this.
@beann You can also make your custom collate function for making a batch of variable sizes.
see this How to create a dataloader with variable-size input
And then you can use adaptive pooling inside the model.