Convnets Ensemble

vision
1

Hey Guys,

I have 3 models I trained and I want to ensemble them together in hope of getting better results.

What is the best way of doing that:

  1. Taking the softmaxes of each model and do averege
  2. Taking the softmaxes and concatenating them and feeding it into Xgboost
  3. Taking the last Linear output (not softmaxes) and feeding it into Xgboost
  4. Other ways I don’t know about

Thanks in advance!

Yana

1 Like
2

How did you train the models? Did you train them on the exact same dataset or random subsets? With same or different hyperparameters?

3

Each model is trained on the exact same dataset not random subsets and with the same hyperparameters the only difference is maybe the learning rate.

Yana

4

I would recommend to retrain them in different subsets and then do a simple majority vote for classification (if you have one-hot coded outputs and else you can average over your predictions) and average over your predictions for regression.
Some approaches use the validation set to learn some weights which are multiplied to each output before summing them up.

1 Like
5

People often use first way as simple. But I guess the second way may better. Have you try the second way