Build Classifier Head Properly


I’m tuning an encoder-decoder model in order to classify images. However I’m a little bit confused about how to build a classifier head. Currently I’m using only an activation function and a Linear layer in the classifier head but I can’t be sure that It’s the way I can only do. How should I build the classifier head properly and how can I be sure about the Linear layer number that I put the classifier head in.