Trying xgboost inside forward , neural net

1 
import torch
import torch.nn as nn
import timm
from xgboost import XGBClassifier
model_name = 'swin_base_patch4_window7_224'

out_dim    = 1


class get_model(torch.nn.Module):
    def __init__(self):
        super().__init__()
        self.model = timm.create_model(model_name, pretrained=False)
        self.model.head = nn.Sequential(nn.Linear(56,768),
                                              nn.Linear(768,256))
        self.last = nn.Linear(256, 128)
        self.depth1 = nn.Linear(128,64)
        self.depth2 = nn.Linear(64,6)
    def forward(self, features,targets):
        x = self.model.head(features)
        x = self.last(x)
        x = self.depth1(x)
        x = self.depth2(x)
        clf = XGBClassifier(tree_method='gpu_hist',
                             num_class = 6,objective = 'multi:softprob')
        clf.fit(x.detach().cpu().numpy(),targets.detach().cpu().numpy())
        preds = clf.predict(x.detach().cpu().numpy())
        preds = torch.tensor(preds)
        return preds

Everything is on GPU … Anything that maybe wrong was this model!! Am i missing something?

2

The error log should show which line is producing the error, you can refer on that.
Also, why detach in the middle of the forward function?

clf.fit(x.detach().cpu().numpy(),targets.detach().cpu().numpy())
preds = clf.predict(x.detach().cpu().numpy())

You know that the back-prop cannot continue from these two lines upward, right?

2 Likes
3

yes! … but didnt thought about that! … do I have to custom the backprop?

4

@duyducvo4444 actually I was trying last return type from XGB … dont think anybody has done it !! yes backprop is problem!

5

I don’t think you can even customize your backprop because the graphs were already detached (i.e. it’s already lost, nothing remains). Backprop needs the graphs to calculate the gradients. That’s why I asked why did you detach in the middle of the forward function.
I think the XGBClassifier you use here is not implemented on tensor, yes?

2 Likes
6

yes… XGB does not support tensor!

7

I have to train … and save the previous layer and return it with the preds and then IG it might work with xgb and saved last layer

8

From the look of it. I think you are trying to train your network to learn the best inputs for XGBClassifier from some features.
But from this line

clf.fit(x.detach().cpu().numpy(),targets.detach().cpu().numpy())

Are you trying to make x to be the same as target?

1 Like
9

Yes… I was trying to learn best inputs for xgb rather than given features and wanted out the outputs of xgb as final outputs…
But think its not possible that way.

10

If you are trying to make x to be the same as target ?
Why don’t you just

def forward(self, features,targets):
        x = self.model.head(features)
        x = self.last(x)
        x = self.depth1(x)
        x = self.depth2(x)
        loss = some_loss_function(x, targets)
        return x, loss

Then back-prop using that loss and do the

clf.predict(x.detach().cpu().numpy())

after the training is done?

2 Likes
11

great!! thanks @duyducvo4444