Having trouble with VOC 2012 segmentation with the void = 255 label

sklearn's confusion matrix should most likely work.
If you would like to fill the confusion matrix in your evaluation loop, have a look at this post.