Hi,
I want to save model predictions for the inputs which are test dataset and its shuffled version, and later use those predictions for calculating accuracy. Because of the inputs, there are 2 predictions for each sample. For calculating accuracy, I want to find those 2 predictions for each sample and getting average over them. How I can do this? I tried to find the same labels and then considering corresponding outputs for them. But achieved accuracy does not make sense.
what about not shuffling?
Without shuffling, I can find them but shuffling is important, and it cannot be removed.
You could write a custom Dataset and return the index together with the data and target and store the index e.g. in a list.
Afterwards you could use these indices to map your samples using your condition.
Thank you very much for your help, @ptrblck.
I saved predictions, targets, indices in the lists. I used for loops to find equal indices and their corresponding predictions, but it is so slow, I wonder is there any ways to speed up finding equal indices?
for i in range(len(prediction_test)):
idx = idx_test[i] #idx:torch.Size([64])
target1 = target_test[i] #target1:torch.Size([64])
prediction1 = prediction_test[i] #prediction1:torch.Size([64, 10])
for j in range(len(idx)):
idx1 = idx[j]
for m in range(len(prediction_test)):
idx2 = idx_testsh[m]
prediction2 = prediction_testsh[m]
for n in range(len(idx2)):
if idx2[n] == idx1:
prediction22 = prediction2[n,:]
break
target = target1[j]
prediction11 = prediction1[j,:]
I’m not completely sure, how all tensors are defined in your code, but I think you could use torch.unique to create a list with all masks and loop once over these masks:
masks = [u==idx for u in idx.unique()]
Would that work for you?
I followed https://discuss.pytorch.org/t/typeerror-dataset-object-does-not-support-indexing/72799/19 for creating the custom datasets, target, and indices. I considered indices for the test dataset and its shuffled version separately, which are called idx_test, idx_testsh. For using torch.unique I put all these indices in one list, but it gives this error:
TypeError: _unique(): argument 'input' (position 1) must be Tensor, not list
How can I apply the output of torch.unique over predictions?
You would have to pass a tensor not a list to torch.unique via e.g. torch.tensor(), or torch.stack/cat.
Thank you for your reply, @ptrblck.
I got that, but the problem is how I can convert a list of tensors to tensor? I tried
idxt = [*idx_test, *idx_testsh]
index = torch.stack(idxt)
it showed this error, (the last batch has size 16 while the rest are 64)
RuntimeError: invalid argument 0: Sizes of tensors must match except in dimension 0. Got 64 and 16 in dimension 1 ...
I also tried
idxt = [*idx_test, *idx_testsh]
i = torch.tensor(idxt)
it gives the error
ValueError: only one element tensors can be converted to Python scalars
torch.cat should work, if you have different batch sizes, but besides that equally sized tensors:
l = [torch.randn(16, 2), torch.randn(64, 2)]
out = torch.cat(l)
torch.cat worked, thank you @ptrblck.
Now I have list of tensors with length 314, and tensors shape is 64x10 except the last one (16x10).
prediction = [*prediction_test, *prediction_testsh]
And the mask with length 10000 and mask[0].shape 20000 :
idxt = [*idx_test, *idx_testsh]
i = torch.cat(idxt)
mask = [u==i for u in torch.unique(i)]
then multiplication between mask and prediction should give the predictions for equal indices? how multiplication between these list of tensors can be done? I tried to convert list of tensors to tensor and then torch.mul(a,b) but that gave error for not matching the size between tensor a and tensor b.
Could you try to loop over these masks and use them to index your predictions?
I tried the loop
for i in range(len(mask)):
prediction_total = mask[i].float().view(-1, 1) * prediction
because of dimension mismatch, I used .view, but the answer is not correct:
>>>print(prediction)
tensor([[3.0603e-02, 5.6551e-02, 9.2520e-02, ..., 4.0616e-02, 1.6195e-01,
4.0573e-02],
[2.9348e-02, 1.0917e-02, 1.8414e-04, ..., 3.0553e-06, 9.5730e-01,
2.0963e-03],
[5.0212e-02, 9.1508e-03, 1.2127e-03, ..., 5.4165e-05, 9.3203e-01,
5.9912e-03],
...,
[1.9600e-02, 1.7821e-01, 2.8131e-03, ..., 7.4272e-03, 2.5914e-02,
7.5438e-01],
[1.4370e-01, 4.3823e-02, 1.7975e-03, ..., 2.7101e-03, 1.4415e-01,
6.6198e-01],
[1.7615e-02, 2.3825e-02, 6.6172e-04, ..., 1.4958e-02, 2.7619e-02,
9.1287e-01]], device='cuda:0')
>>>print(mask[0])
tensor([1, 0, 0, ..., 0, 0, 0], device='cuda:0', dtype=torch.uint8)
for i=0 >>>print(prediction_total)
tensor([[0.0306, 0.0566, 0.0925, ..., 0.0406, 0.1619, 0.0406],
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
...,
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000],
[0.0000, 0.0000, 0.0000, ..., 0.0000, 0.0000, 0.0000]],
device='cuda:0')
prediction_total supposed to be nonzero just in two values, but it is nonzero for the whole first row.