Huge False Negatives

7

below is the code:

weight = [3.17]
class_weight = torch.FloatTensor(weight)
loss_function = nn.BCEWithLogitsLoss(pos_weight=class_weight)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

y_pred = model(categorical_train_data, numerical_train_data)
print(y_pred.shape, train_outputs.shape)
single_loss = loss_function(y_pred, train_outputs)
#single_loss = loss_function(y_pred, torch.max(train_outputs,1)[1])

output:

torch.Size([39074, 2]) torch.Size([39074, 1])
  File "/apps/miniconda3/envs/torch/lib/python3.8/site-packages/torch/nn/functional.py", line 2124, in binary_cross_entropy_with_logits
raise ValueError("Target size ({}) must be the same as input size ({})".format(target.size(), input.size()))
 ValueError: Target size (torch.Size([39074, 1])) must be the same as input size (torch.Size([39074, 2]))

Just checking what to do

8

what should i do? should i reduce my input y_pred to 1 dimension ?

9

Based on the target shape it seems you are dealing with a binary classification.
If you are using nn.BCEWithLogitsLoss, then your model would have to output a single logit, while it seems to be returning currently 2 outputs.

10

okay… is it the same scenario in multi-classification too ? if you have 0-9 then output will be single logit only.

11

this blog tutorial also showed binary classification, but they have updated output_size as 2, does they mentioned wrong ?

12

if the things mentioned in blog is true, should i go for opt the solution 2 mentioned in below site (reshape the input tensor from n.m to n.l)

http://www.programmersought.com/article/1886850916/

13

For a binary classification you could use:

  • one output value and use nn.BCEWithLogitsLoss
  • or two outputs and use nn.CrossentropyLoss

Both approaches would work, but are a bit different in their implementation.
While the former is used for a classical binary classification as well as a multi-label classification (each sample might belong to zero, one, or multiple classes), the latter is used for a multi-class classification (each sample belongs to one class only).

14

Okay, Thanks i will try to BCEWithLogitsLoss as mine in binary classification, will try to update output_size as 1 and will try to change obtained tensor from model to int from float as target is type of int.

15

Hi @ptrblck,

tried all the possibilities, even followed the suggestion given by @done1892 from What is the weight values mean in torch.nn.CrossEntropyLoss? - #10 by done1892 for BCEwithlogitsloss.

This is my classification ratio in training set.

<=50K 29705
(greater than)50K 9369
Name: class, dtype: int64

so i have tried with pos_ratio of 3.17 and 1/3.17 too… but still none of the confusion_matrix or accuracy didnt changed a bit.

Here is my code:

weights = [[2.5],[2.55],[2.6],[2.65],[2.7],[2.75],[2.8],[2.85],[2.9],[2.95],[3],[3.05],[3.1],[3.15],[3.2],[3.25],[3.3],[3.35],[3.4],[3.45],[3.5],[3.55],[3.6],[3.65],[3.7],[3.75],[3.8],[3.85],[3.9],[3.94999999999999],[3.99999999999999],[4.04999999999999],[4.09999999999999],[4.14999999999999],[4.19999999999999],[4.24999999999999],[4.29999999999999],[4.34999999999999],[4.39999999999999],[4.44999999999999],[4.49999999999999],[4.54999999999999],[4.59999999999999],[4.64999999999999],[4.69999999999999],[4.74999999999999],[4.79999999999999],[4.84999999999999],[4.89999999999999],[4.94999999999999],[4.99999999999999],[5.04999999999999],[5.09999999999999],[5.14999999999999],[5.19999999999999],[5.24999999999999],[5.29999999999999],[5.34999999999999],[5.39999999999999],[5.44999999999999],[5.49999999999999],[5.54999999999999],[5.59999999999999],[5.64999999999999],[5.69999999999999],[5.74999999999999],[5.79999999999999],[5.84999999999999],[5.89999999999999],[5.94999999999999],[5.99999999999999],[6.04999999999999],[6.09999999999999],[6.14999999999999],[6.19999999999999],[6.24999999999999],[6.29999999999999],[6.34999999999999],[6.39999999999999],[6.44999999999999],[6.49999999999999],[6.54999999999999],[6.59999999999999],[6.64999999999999],[6.69999999999999],[6.74999999999998],[6.79999999999998],[6.84999999999998],[6.89999999999998],[6.94999999999998],[6.99999999999998],[7.04999999999998],[7.09999999999998],[7.14999999999998],[7.19999999999998],[7.24999999999998],[7.29999999999998],[7.34999999999998],[7.39999999999998],[7.44999999999998],[7.49999999999998],[7.54999999999998],[7.59999999999998],[7.64999999999998],[7.69999999999998],[7.74999999999998],[7.79999999999998],[7.84999999999998],[7.89999999999998],[7.94999999999998],[7.99999999999998],[8.04999999999998],[8.09999999999998],[8.14999999999998],[8.19999999999998],[8.24999999999998],[8.29999999999998],[8.34999999999998],[8.39999999999998],[8.44999999999998],[8.49999999999998],[8.54999999999998],[8.59999999999998],[8.64999999999998],[8.69999999999998],[8.74999999999998],[8.79999999999998],[8.84999999999998],[8.89999999999998],[8.94999999999998],[8.99999999999998],[9.04999999999998],[9.09999999999998],[9.14999999999998],[9.19999999999998],[9.24999999999998],[9.29999999999998],[9.34999999999998],[9.39999999999998],[9.44999999999998],[9.49999999999998],[9.54999999999997],[9.59999999999997],[9.64999999999997],[9.69999999999997],[9.74999999999997],[9.79999999999997],[9.84999999999997],[9.89999999999997],[9.94999999999997],[9.99999999999997],[10.05],[10.1],[10.15],[10.2],[10.25],[10.3],[10.35],[10.4],[10.45],[10.5],[10.55],[10.6],[10.65],[10.7],[10.75],[10.8],[10.85],[10.9],[10.95],[11],[11.05],[11.1],[11.15],[11.2],[11.25],[11.3],[11.35],[11.4],[11.45],[11.5],[11.55],[11.6],[11.65],[11.7],[11.75],[11.8],[11.85],[11.9],[11.95],[12],[12.05],[12.1],[12.15]
]

rate = [0.001,0.005,0.009,0.013,0.017,0.021,0.025,0.029,0.033,0.037,0.041,0.045,0.049,0.053,0.057,0.061,0.065,0.069,0.073,0.077]

for lr in rate:
	for wg in weights:
		#Create Model object
		model = Model(categorical_embedding_sizes, numerical_data.shape[1], 1, [512,128,16], p=0.4)
		
		#Create loss and optimizer function

		#Add Weights
		#weights = [3.17,1.0]
		#class_weights = torch.FloatTensor(weights)

		#loss_function = nn.CrossEntropyLoss(weight=class_weights)
		class_weight = torch.FloatTensor(wg)
		loss_function = nn.BCEWithLogitsLoss(pos_weight=class_weight)
		optimizer = torch.optim.Adam(model.parameters(), lr=0.001)



		#Train the model

		epochs = ep
		aggregated_losses = []

		for i in range(epochs):
			i += 1
			y_pred = model(categorical_train_data, numerical_train_data)
			#print(y_pred[:5])
			#print(y_pred.shape, train_outputs.shape)
			#out, inds = torch.max(y_pred,dim=1)
			#out = out.unsqueeze(1)
			train_outputs = train_outputs.float()
			single_loss = loss_function(y_pred, train_outputs)
			#single_loss = loss_function(y_pred, torch.max(train_outputs,1)[1])
			aggregated_losses.append(single_loss)

			#if i%25 == 1:
			#	print(f'epoch: {i:3} loss: {single_loss.item():10.8f}')

			optimizer.zero_grad()
			single_loss.backward()
			optimizer.step()

		print(f'epoch: {i:3} loss: {single_loss.item():10.10f}')


		#plt.plot(range(epochs), aggregated_losses)
		#plt.ylabel('Loss')
		#plt.xlabel('epoch');




		##Make Predictions
		with torch.no_grad():
			y_val = model(categorical_test_data, numerical_test_data)
			#out, inds = torch.max(y_val,dim=1)
			#out = out.unsqueeze(1)
			test_outputs = test_outputs.float()
			loss = loss_function(y_val, test_outputs)
			#loss = loss_function(y_val, torch.max(test_outputs,1)[1])
		print(f'Validation Loss: {loss:.8f}')


		#Convert output values to either 0 or 1
		y_val = np.argmax(y_val, axis=1)
		#print(y_val[:5])


		print(confusion_matrix(test_outputs,y_val))
		#print(classification_report(test_outputs,y_val))
		print(accuracy_score(test_outputs, y_val))

please take a look and advice.

16

FYI
i tried with catboost and with simple tuning i am able to achieve good results…i dont understand why this much difficult with DNN using pytorch.

    Test data:
 <=50K    7451
 >50K     2318
Name: class, dtype: int64
 Confusion matrix:
[[7037  799]
 [ 414 1519]]

Here we got 799 FN’s and 414 FP’s

17

@ptrblck or any one suggest please

18

Hi,

even weights are not helping out for imbalanced data… please let me know how to do it.

19

Using a weighted loss usually trades the metrics, e.g. FP for FN.
Are you not seeing any change in the confusion matrix using a weighted loss?

20

no, not at all. wondered to see no changes… as per thumb rule i have assigned weight (lowest/highest)

weight = [0.317]
class_weight = torch.FloatTensor(weight)
loss_function = nn.BCEWithLogitsLoss(pos_weight=class_weight)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)

Here is the classification groupings:

traindata:

          <=50K    29705
          >50K      9369

testdata:

 <=50K    7450
 >50K     2318

assigned weight as (9369/29705) and got CM as it is as if no weights are assigned:

epoch: 300 loss: 0.3204252720
Validation Loss: 0.31998613
[[7450    0]
 [2318    0]]

			  precision    recall  f1-score   support

		 0.0       0.76      1.00      0.87      7450
		 1.0       0.00      0.00      0.00      2318

	accuracy                           0.76      9768
   macro avg       0.38      0.50      0.43      9768
weighted avg       0.58      0.76      0.66      9768

0.7626945126945127
21

Assuming class1 is positive and class0 negative, shouldn’t the weight be passed as the reciprocal?

From the docs:

For example, if a dataset contains 100 positive and 300 negative examples of a single class, then pos_weight for the class should be equal to 300/100=3. The loss would act as if the dataset contains 3 * 100=300 positive examples.

22

yeah tried reciprocal too… same results…

23

Could you try increasing it until the confusion matrix changes?

24

ok give some suggestions… anyway i will keep some for loop… what ranges should i give ?

25

Hi @ptrblck,

below are the weights i tested and there is no deviation in confusion matrix and one more observation is as weight value is increasing difference between validation loss and training loss is increasing. so is that i need to test for lower weights ? and what is the weight difference i should keep in for loop ? (value of 10 ?)

[[2.5],[2.55],[2.6],[2.65],[2.7],[2.75],[2.8],[2.85],[2.9],[2.95],[3],[3.05],[3.1],[3.15],[3.2],[3.25],[3.3],[3.35],[3.4],[3.45],[3.5],[3.55],[3.6],[3.65],[3.7],[3.75],[3.8],[3.85],[3.9],[3.94999999999999],[3.99999999999999],[4.04999999999999],[4.09999999999999],[4.14999999999999],[4.19999999999999],[4.24999999999999],[4.29999999999999],[4.34999999999999],[4.39999999999999],[4.44999999999999],[4.49999999999999],[4.54999999999999],[4.59999999999999],[4.64999999999999],[4.69999999999999],[4.74999999999999],[4.79999999999999],[4.84999999999999],[4.89999999999999],[4.94999999999999],[4.99999999999999],[5.04999999999999],[5.09999999999999],[5.14999999999999],[5.19999999999999],[5.24999999999999],[5.29999999999999],[5.34999999999999],[5.39999999999999],[5.44999999999999],[5.49999999999999],[5.54999999999999],[5.59999999999999],[5.64999999999999],[5.69999999999999],[5.74999999999999],[5.79999999999999],[5.84999999999999],[5.89999999999999],[5.94999999999999],[5.99999999999999],[6.04999999999999],[6.09999999999999],[6.14999999999999],[6.19999999999999],[6.24999999999999],[6.29999999999999],[6.34999999999999],[6.39999999999999],[6.44999999999999],[6.49999999999999],[6.54999999999999],[6.59999999999999],[6.64999999999999],[6.69999999999999],[6.74999999999998],[6.79999999999998],[6.84999999999998],[6.89999999999998],[6.94999999999998],[6.99999999999998],[7.04999999999998],[7.09999999999998],[7.14999999999998],[7.19999999999998],[7.24999999999998],[7.29999999999998],[7.34999999999998],[7.39999999999998],[7.44999999999998],[7.49999999999998],[7.54999999999998],[7.59999999999998],[7.64999999999998],[7.69999999999998],[7.74999999999998],[7.79999999999998],[7.84999999999998],[7.89999999999998],[7.94999999999998],[7.99999999999998],[8.04999999999998],[8.09999999999998],[8.14999999999998],[8.19999999999998],[8.24999999999998],[8.29999999999998],[8.34999999999998],[8.39999999999998],[8.44999999999998],[8.49999999999998],[8.54999999999998],[8.59999999999998],[8.64999999999998],[8.69999999999998],[8.74999999999998],[8.79999999999998],[8.84999999999998],[8.89999999999998],[8.94999999999998],[8.99999999999998],[9.04999999999998],[9.09999999999998],[9.14999999999998],[9.19999999999998],[9.24999999999998],[9.29999999999998],[9.34999999999998],[9.39999999999998],[9.44999999999998],[9.49999999999998],[9.54999999999997],[9.59999999999997],[9.64999999999997],[9.69999999999997],[9.74999999999997],[9.79999999999997],[9.84999999999997],[9.89999999999997],[9.94999999999997],[9.99999999999997],[10.05],[10.1],[10.15],[10.2],[10.25],[10.3],[10.35],[10.4],[10.45],[10.5],[10.55],[10.6],[10.65],[10.7],[10.75],[10.8],[10.85],[10.9],[10.95],[11],[11.05],[11.1],[11.15],[11.2],[11.25],[11.3],[11.35],[11.4],[11.45],[11.5],[11.55],[11.6],[11.65],[11.7],[11.75],[11.8],[11.85],[11.9],[11.95],[12],[12.05],[12.1],[12.15]
]

26

hi @ptrblck @mrshenli

weights didnt worked out. can you please help me out to get ~99% accuracy…