Features Selection

I want to use Fisher score to select two model’s feature. One is resnet34, another is resnet50. I ran the program a few times but got very bad result. The accuracy is about 3%. I don’t know what’s wrong:sob::sob:Hear is my every epoch in model train

input = input.cuda()
target = target.cuda()
input_var = torch.autograd.Variable(input)
target_var = torch.autograd.Variable(target)
#model1 is resnet34 and I removed its classifier(nn.Linear())
output1 = model1(input_var, target_var)#(its shape is[batch_size, 512])
#model2 is resnet50 and I removed its classifier too(nn.Linear())
output2 = model2(input_var, target_var)#(its shape is[batch_size, 2048])
output = torch.cat((output1, output2), 1)#(its shape is[batch_size, 2560])
output = selected_feautres(output, target)#(its shape is[batch_size, 1280])
classifier inn.Linear(1280, num_class).cuda()
output = classifier(output)#(its shape is[batch_size, num_class=35])
loss = ...

The code of selected_output is following

def selected_feautres(x, target):
    tmp = x
    tmp = tmp.data.cpu().numpy()
    x1 = x
    x1 = x1.data.cpu().numpy()
    target1 = target
    target1 = target1.data.cpu().numpy()
    #compute 2560 features's fisher score
    x1 = fisher_score(x1, target1)
    Rank features in descending order according to fisher score,
    the larger the fisher score, the more important the feature is
    x1 = feature_ranking(x1)
    number = 1280
    x1 = list(x1[0:number])
    x = tmp[:, x1]
    x = torch.from_numpy(x).cuda()
    return x

The fisher_score and feature_ranking is from the following github