Is it possible to ignore masked part of sequences when creating an encoder?

Hello! I am attempting to make an observation history encoder, where my goal is for a model that takes in as input a variable length sequence of dimension [Time, Batch, Features] (where sequences are padded to fit a fixed Time length) and outputs a variable of dimension [Batch, New_Features]. My concern is that when I’m doing dimensionality reduction with FC layers, they will take the padded data into consideration. Is there any way to avoid this?