#model1
def getmodel():
model = EfficientNet.from_pretrained('efficientnet-b5')
model._fc = nn.Sequential(
nn.Linear(in_features=2048, out_features=1024, bias=True), #first layer
nn.ReLU(),
nn.BatchNorm1d(1024, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True),
nn.Dropout(p=0.5),
nn.Linear(in_features=1024, out_features=4, bias=True)) #last layer
model = model.to(device)
return model
I have lot of question from the above code block which I want to clarify.
- I have read that bias should be True (bias=True) at the last linear layer. And my model also performed well when turned on.
- Most people suggested that bias should be turned off (bias=False) before using batch norm ( Even bias in the Conv layers of EfficientNet are turned off before batch norm). But my model performed badly when I turned off the bias at the first layer. What should I follow?
- This model (model2) just uses the single linear layer.
#model2
def getmodel():
model = EfficientNet.from_pretrained('efficientnet-b5')
model._fc = nn.Sequential(
nn.Linear(in_features=2048, out_features=4, bias=True)) #last layer
model = model.to(device)
return model
Whcih approch should I follow model1 or model2 ?