Please give me some advice on model building.

Hi.

I am in the process of building an original model based on Faster-R CNN. Specifically, it is a model that outputs the attributes of a class in addition to outputting the class and boundary box.As an image, I thought it should look like the figure below. I would like some advice. Thanks!