What is the best way to use golden feature in machine learning model?

Let’s say I have a classification model. And my job is to predict the correct class out of 30 different classes. The current accuracy is 70%.

The thing is: I have to consume another team’s classification result which is 80% accurate. So I’m using their prediction result as a feature. I’ll call it “golden feature”. Let’s say I’m aiming >80% accuracy with the golden feature.

Here is my current approach:

(I’m using Deep Learning.) I have several features and each feature has its own weight. I also create a weight vector for one hot vector (1 by 30) of “golden feature” and train all weights together. However the result doesn’t seem to provide much.

I thought about the reason why and realized that the learned vector (30 by n) won’t be that meaningful. They would be just positive numbers.
(Please yell at me if my reasoning is wrong!)

Has anyone faced the similar problem? Any suggestion will be highly appreciated!
The method that you suggest doesn’t have to be Deep Learning approach.

You could try to use the output of the penultimate layers, concatenate them, and train a final classifier. This would be similar to the vanilla fine tuning approach, but with multiple inputs.

If this doesn’t provide much benefit, I would go for e.g. XGBoost and try to create a model ensemble out of your trained models.

1 Like