I have batched and sequenced 2d data, meaning I am working with tensors of size (batch_size, max_seq_len, d1, d2). I want to use convolution to reduce it to (batch_size, max_seq_len, new_dim). The layer that does this doesn’t care about sequences, meaning all elements of the sequence are passed through the same weights. The issue is that Conv1d and Conv2d do not accept 4d inputs, so I would have to first reshape them into (batch_size * max_seq_len, d1, d2).
My question is, is there a more efficient way to do this? Or is it ok? The worry is that batch_size*max_seq_len ends up equalling 2400, which seems to big for a batch and it might cause my computation to run too slowly.