I am working to improve the quality of a user model. The changes that I am making makes the results worth for new users.
We already are over predicting for new users and currently don’t have a solution.
My main question is regarding these new users, how can I makes the prediction better for those, or at least push it towards under prediction ( we pay more out of pocket when we over predict).
Model is 5 layer relus with last layer being sigmoid.
Not sure if this is the right forum. Thanks.
Could you elaborate more about the use case (business requirement)? type of data? What EDA steps have you used?
And since you have just mentioned about over-fitting? Have you tried either decreasing your model complexity or some regularization techniques?
I tried to keep some of the business information intentionally.
The goal is to have better prediction for our customers. For example assume the prediction is the number of clicks need to be predicted.
The model is already well established and I haven’t done much EDA other than seeing that when slice by the age of my customer on my website (let’s say users that are 5 days on my website compare to users that have a lot of activities in the past 5 years). I see that the prediction for these users is really bad. We do use user IDs in our models and other identifying information.
Some people call this a cold start problem but it is mostly in ads space I am not sure how to apply it here.
Re over fitting, I don’t think my model is over fitting. Rather when I look at the calibration graphs, we are consistently over predicting for new users compared to the real number clicks they make.