Keypoints RCNN multiple classes with different number of keypoints


I’m interested in creating a Keypoints RCNN detecting objects and keypoints for two classes: pants and t-shirts. There are different number of keypoints associated with each class (7 and 10). What I tried is to first train for separate classes (one model for pants and one for t-shirts) and was able to succeed. However, I wonder if it’s possible to do this together. My approach here was to simply create an array big enough to store both numbers of keypoints (17) and simply set first 7 to 0 and not visible, then proper keypoints locations. Similar thing for pants, where first 7 points have correct coordinates and the rest is 0s with not visible. Unfortunately I wasn’t able to train any working model, even for a very small dataset (20 examples). Is it possible to approach this problem in another way?