How to properly handle inference cancelation in a TorchServe handler?

Hi all,

I’ve been wondering about the proper way of handling and stopping inference if a client cancels their request. My use case is that I have a custom TorchServe handler, which implements the initialize, preprocess, inference, and postprocess methods. The process takes a while - maybe around 50-60 seconds, so I thought it would be a good idea to handle any cancelations by interrupting the inference progress so that further requests can continue faster.

Is there any method, which I can implement to give me details about the underlying connection, such as when it was forcefully closed by the client?

Thanks,
Miro