Size of LSTM output and linear input

Hi, I have a question when using LSTM,
the input size of LSTM is [4,256,100], so this is a 4 seconds sequence, each one has length 100, and batch size is 256. and the output of the LSTM is [4,256,20].
I added a Linear layer after the LSTM, the size of linear is (425620,output_size).
But during testing, the input will be [4,1,100], which is only one batch, and there will be an error for the input size of the Linear layer. Are there any ways to fix this?

I know in CNN, the batch size for training and testing can be different, during training, I set the batch size to be 32 or 64, and during testing, I can set the batch size to be 1 and it still works