No gradient When loss is calculated using a coordinate value in the segmentation map

A predicted segmentation map (sigmoid output), I want to make constraints on the minimum peripheral box of the segmented object, but directly calculate the peripheral box coordinates will lead to no gradient, how to preserve this gradient, and optimize the segmentation mask by optimizing the peripheral box coordinates?