The number of pictures in the confusion matrix is far less than the actual number of pictures in the test set. What is going on?
Your code might miss some test predictions. 
Without seeing any code, I can just speculate, so please feel free to post the code used to calculate the confusion matrix.