Bad Inference result during testing even when using training data

1

Hi, I have encounter a weird behavior from my model, during training phase, I printed out some of my validation result from my model and the result is very satisfying but after I stop my training and try to do testing with it, I got a very bad result which is very different from the validation result, even when I am using data used for training to do the testing. Here is some of my validation result:

Test_8_0_recon
Test_8_0_recon2066×2066 810 KB
.
While this is some of the result I got from testing, only pay attention to the color distribution:
Screenshot from 2020-03-21 08-53-22
Screenshot from 2020-03-21 08-53-22498×509 141 KB

as you can see, the result is grayish not even close to how the validation result looks like. So I am hoping someone who knows what is the problem with my model can give me some clue to fix this.

2

Are you calling model.eval() for the validation and test case?
Also, make sure to use the same preprocessing for both use cases.

If that doesn’t help, could you post your training and validation/testing loop, so that we could have a look?

3

During validation I use with torch.no.grad() before starting , while for test case I use model.eval() and yes I use the same pre-processing steps, the only difference would be the validation use batch size of 64 while during testing I use batch size of 1 since I use different PC for them. Would that be a problem?

4

Could you use model.eval() for both, the validation and testing phase?
This will make sure to disable dropout as well as use the running stats of batchnorm layers instead of the current batch stats.

If you validation results are bad after using model.eval(), then the batchnorm stats might be off.

PS: torch.no_grad() is still useful for both phases, as it will save memory.

5

Hi, thanks for the reply. Apparently after discussing with my friend, we found out that the problem lies on my model. But thank you for your advice on model_eval(), I will use it on my future work

6

Hi luminoussin, how did you solve this issue, because i am also facing same issue, and regarding model.evaI() i pretty sure about that, I am calling this eval() from my prediction function but prediction result is not good.

7

Hi Amiya, as I said on my previous post the problem turn out to be on my model itself. So if you already use model.eval() before doing inference with your trained model then that means there is a possibility that your model has overfitted. At least that was what happened on mine

8

Hi luminoussin, can you check my code? i am in beginner level.

9

I think it is better if you create your own post about it. Therefore more people can help you and it is better if you post your code instead of giving link

10

I had same problem and in my case it was due to loading images, I needed to convert my images and rescale them to (32x32x1)

img = cv2.imread(path)
img = cv2.cvtColor(img, cv2_COLORBGR2GRAY)`
img = cv2.resize( img, (32,32) )

but when I change it to

transform_valid = transforms.Compose([
    transforms.ToPILImage(), 
    transforms.Resize((32, 32)),
    transforms.ToTensor()])

img = cv2.imread(path, 0)
img = custom_transfrom(img).unsqueeze(0)

my problem solved