I had to convert my CLDNN model from Keras to Torch. for work reasons.
In order to restore the performance of the model on Keras, the model structure, hy-params,the initialization of weight and bias are still done addcording to my CLDNN model.
The current result is that the classifcation result of torch is 3% less than the result on keras. The performance degardation is fatal in terms of my work.
Is it possible that you are overlooking some small error? Like I was not calling model.eval() couple days back, and It was giving around 5% more error. Please verify.