My dataset consists of arrays whose elements are between -1.5 and 2.0. But in these arrays there are some elements which are -10. I want to train my model ( image-to-image prediction) without considering these -10 elements.
One of my arrays is looking like bottom one( as a representation).
Blockquote
Is there any way to filter this data by giving threshold or any other way so that in the output array from my model preserves these -10s and predicts other data only.
Can’t you make it as zero ? Like x(x==-10) = 0
@Rakuen
Applying filters within your model would be difficult. Instead you can apply filters on the input data like
x[x!=-10]
while sending data to the model to be trained
The problem is foresee is that during prediction(inference) you would have to apply the same rule, so that your model does not get to predict the rows for -10
Thank you. I will try but I didn’t understand it all. Ok, I can send data to model without-10s with your suggestion but I am not sure whether my model gives an error about its size. Anyway I will try and see tomorrow. I also thought that I can use something like torch.clamp(output, min=-9) but didn’t try it neither.
As I expected, It didn’t work. Size is the problem. Data has size
torch.Size([1, 1, 256, 256])
but after x[x!=-10]
torch.Size([62779])
So my model didn’t accept data. But I also tried torch.clamp. I applied this to my data before sending to model and for the ground truth before calculating loss function and so backpropagation. It is working but in a wrong way. I want to estimate weights of my model without taking account of this -10 data. Because I think my model predicts near this -10 region by considering it.
If I understand correctly, you want your input to keep the same size (256 x 256)? What does your ground-truth looks like? (not super familiar with image-to-image prediction)
Yes, I want to keep it the same (256x256) and my model will be trained for just non-10 data:)
Ok, what loss function do you use? And are your targets (ground-truth) also images?
MSE. Inputs and targets are numpy arrays (256x256, float).
Ok, here what I think you could try. Basically, your model will learn to not predict the -10 values and hopefully learn to ignore them from the input. It may not be exactly as you want, but it is the only way I can see it done without modifying the size of the input. It is inspired by what is done in stereo estimation, as you can see here code.
# consider only the elements different of -10
mask = (target != -10)
mask.detach_()
...
loss = F.mse_loss(input[mask], target[mask])
loss.backward()
...
Ok, I will try. Thank you.