Follow this, it is done for semantic segmentation. I think a similar thing can be worked out for your case.