Hello,
Thanks for your response.The problems with this model behavior was related to BatchNorm. I forgot to switch model to eval mode. This switching helps me to resolve this problem.
This problem was discussed here too:
https://discuss.pytorch.org/t/output-varies-when-changing-batch-size-during-test/10439