This is only my first day posting on this forum, though I have visited many times over the past 3+ years of using Pytorch, and benefited from the questions and answers here. I apologize for the long post below.
I am fine-tuning Keypoint RCNN on my custom dataset and, using resources I have found in various places, am successfully training and obtaining bounding box and keypoint predictions on multiple classes together with very good results. Due to my use case, I am in need of employing the keypoint visibility flag which the model is meant to support. According to the model’s github page, this is supposed to be the same flag convention as the COCO dataset (v=0 ‘not in image’; v=1 ‘occluded’; v=2 ‘visible’) but it turns out that Keypoint RCNN does not strictly follow this convention in practice. Based on my results and those of other users I have encountered on different forums and sites, Keypoint RCNN always predicts a flag value of v=1 for all keypoints, no matter the training flag v>0 (even v=0), and predicts coordinates for them as well. From what I can gather the model is supposed to exclude from loss calculation any keypoints with flag v=0, which it may be doing. But still goes on to predict v=1 and nonzero coordinates for them later. I have tried training such keypoints using coordinates (0.0,0.0) as found in COCO but with no success.
My hope in writing here is to get some help in tracking down the source code where Keypoint RCNN is converting all flags to v=1 and handling flags so that I might be able to modify this behavior, as the model does not seem to learn the flag values presently. In my use case, what I want is for Keypoint RCNN to successfully predict the right flag (e.g. v=0) so that I can use it later on, or at least predict a coordinate of (0.0,0.0) (or some other fixed value) for keypoints with v=0. The need is to be able to distinguish between visible and occluded keypoints. Even just two learned flags that work as expected (v=0 and v=1) would be very useful to have. Any suggestions or guidance would be great. I wonder if it is reasonable to make such a request on the github page, hoping that the feature can be added to the model? Thanks for taking the time to reply.
For anyone interested in an update on this topic, please see the below link for a github conversation I started with the Pytorch developers regarding the keypoint visibility flag behavior of Keypoint RCNN. Some additional helpful information can be found there. Hopefully, further dialogue will continue, and maybe even with progress on a solution.
